[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 479779522 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003314] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.007870] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010134] LSM: Security Framework initializing [ 0.011057] Yama: becoming mindful. [ 0.012039] SELinux: Initializing. [ 0.013071] *** VALIDATE selinux *** [ 0.021809] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025711] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026161] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028133] *** VALIDATE tmpfs *** [ 0.029477] *** VALIDATE proc *** [ 0.030249] *** VALIDATE cgroup *** [ 0.031013] *** VALIDATE cgroup2 *** [ 0.032263] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033166] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036011] Spectre V2 : User space: Vulnerable [ 0.037010] Speculative Store Bypass: Vulnerable [ 0.039933] debug: unmapping init [mem 0xffffffffad059000-0xffffffffad060fff] [ 0.041859] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042661] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043024] ... version: 2 [ 0.044015] ... bit width: 48 [ 0.045014] ... generic registers: 4 [ 0.046012] ... value mask: 0000ffffffffffff [ 0.047015] ... max period: 00007fffffffffff [ 0.048015] ... fixed-purpose events: 3 [ 0.049013] ... event mask: 000000070000000f [ 0.050313] rcu: Hierarchical SRCU implementation. [ 0.052529] smp: Bringing up secondary CPUs ... [ 0.053600] x86: Booting SMP configuration: [ 0.054029] .... node #0, CPUs: #1 #2 #3 [ 0.057510] smp: Brought up 1 node, 4 CPUs [ 0.059014] smpboot: Max logical packages: 1 [ 0.060020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.139431] node 0 deferred pages initialised in 77ms [ 0.143097] devtmpfs: initialized [ 0.144283] x86/mm: Memory block size: 128MB [ 0.148172] gcov: version magic: 0x41383552 [ 0.152331] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156145] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.159520] pinctrl core: initialized pinctrl subsystem [ 0.161280] [ 0.161811] ************************************************************* [ 0.164025] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167020] ** ** [ 0.169019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171023] ** ** [ 0.173019] ** This means that this kernel is built to expose internal ** [ 0.176022] ** IOMMU data structures, which may compromise security on ** [ 0.178034] ** your system. ** [ 0.180020] ** ** [ 0.183018] ** If you see this message and you are not debugging the ** [ 0.185019] ** kernel, report this immediately to your vendor! ** [ 0.187022] ** ** [ 0.189016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.191018] ************************************************************* [ 0.194514] NET: Registered protocol family 16 [ 0.195455] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.198072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.201067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.205112] cpuidle: using governor menu [ 0.206612] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.208608] PCI: Using configuration type 1 for base access [ 0.211109] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.219197] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.221014] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.224068] cryptd: max_cpu_qlen set to 1000 [ 0.226858] ACPI: Added _OSI(Module Device) [ 0.228010] ACPI: Added _OSI(Processor Device) [ 0.229010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.230009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.234612] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.239375] ACPI: Interpreter enabled [ 0.240057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.240953] ACPI: Using IOAPIC for interrupt routing [ 0.242099] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.245271] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.253362] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.255041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.257018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.259061] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.263365] acpiphp: Slot [2] registered [ 0.264000] acpiphp: Slot [5] registered [ 0.266127] acpiphp: Slot [6] registered [ 0.267031] acpiphp: Slot [7] registered [ 0.268066] acpiphp: Slot [8] registered [ 0.269105] acpiphp: Slot [9] registered [ 0.270107] acpiphp: Slot [10] registered [ 0.271087] acpiphp: Slot [3] registered [ 0.272027] acpiphp: Slot [4] registered [ 0.272968] acpiphp: Slot [11] registered [ 0.274144] acpiphp: Slot [12] registered [ 0.275124] acpiphp: Slot [13] registered [ 0.277133] acpiphp: Slot [14] registered [ 0.278116] acpiphp: Slot [15] registered [ 0.279153] acpiphp: Slot [16] registered [ 0.281133] acpiphp: Slot [17] registered [ 0.284202] acpiphp: Slot [18] registered [ 0.285105] acpiphp: Slot [19] registered [ 0.287157] acpiphp: Slot [20] registered [ 0.289191] acpiphp: Slot [21] registered [ 0.290127] acpiphp: Slot [22] registered [ 0.291125] acpiphp: Slot [23] registered [ 0.293151] acpiphp: Slot [24] registered [ 0.294130] acpiphp: Slot [25] registered [ 0.296235] acpiphp: Slot [26] registered [ 0.298107] acpiphp: Slot [27] registered [ 0.299176] acpiphp: Slot [28] registered [ 0.301172] acpiphp: Slot [29] registered [ 0.302119] acpiphp: Slot [30] registered [ 0.304108] acpiphp: Slot [31] registered [ 0.305108] PCI host bridge to bus 0000:00 [ 0.306042] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.309030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.311021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.314049] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.315060] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.317032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.318159] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.319853] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.321956] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.331013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.336549] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.339028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.341026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.343031] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.346559] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.349835] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.352045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.355011] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.361016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.373018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.379017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.385287] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.395027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.402024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.417024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.428355] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.441017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.452018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.473067] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.486927] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.493029] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.504026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.518025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.529432] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.542020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.551022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.578022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.594499] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.600021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.606025] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.620025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.629884] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.635021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.641022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.655025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.666168] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.668483] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.670480] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.673394] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.676366] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.681035] iommu: Default domain type: Passthrough [ 0.682609] SCSI subsystem initialized [ 0.683134] ACPI: bus type USB registered [ 0.684192] usbcore: registered new interface driver usbfs [ 0.686136] usbcore: registered new interface driver hub [ 0.687109] usbcore: registered new device driver usb [ 0.689227] pps_core: LinuxPPS API ver. 1 registered [ 0.690013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.693098] PTP clock support registered [ 0.695268] EDAC MC: Ver: 3.0.0 [ 0.697641] PCI: Using ACPI for IRQ routing [ 0.700915] NetLabel: Initializing [ 0.702011] NetLabel: domain hash size = 128 [ 0.704013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.706112] NetLabel: unlabeled traffic allowed by default [ 0.708112] vgaarb: loaded [ 0.709318] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.711017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.717664] clocksource: Switched to clocksource kvm-clock [ 0.832775] VFS: Disk quotas dquot_6.6.0 [ 0.834104] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.836234] *** VALIDATE ramfs *** [ 0.837626] *** VALIDATE hugetlbfs *** [ 0.839384] pnp: PnP ACPI init [ 0.842169] pnp: PnP ACPI: found 6 devices [ 0.862416] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.866117] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.868522] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.870811] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.873410] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.875909] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.879028] NET: Registered protocol family 2 [ 0.881558] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.886729] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.890460] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.896238] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.899726] TCP: Hash tables configured (established 65536 bind 65536) [ 0.903143] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.906707] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.910392] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.913309] NET: Registered protocol family 1 [ 0.915848] RPC: Registered named UNIX socket transport module. [ 0.918081] RPC: Registered udp transport module. [ 0.919739] RPC: Registered tcp transport module. [ 0.921588] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.923945] NET: Registered protocol family 44 [ 0.925664] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.927553] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.929793] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.932340] PCI: CLS 0 bytes, default 64 [ 0.933973] Unpacking initramfs... [ 2.332802] debug: unmapping init [mem 0xffff92a6bcc54000-0xffff92a6bffbffff] [ 2.336693] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.339361] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.342444] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.861174] Initialise system trusted keyrings [ 2.862980] Key type blacklist registered [ 2.864750] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.877608] zbud: loaded [ 2.881504] *** VALIDATE nfs *** [ 2.882848] *** VALIDATE nfs4 *** [ 2.884798] pstore: using deflate compression [ 2.888737] Platform Keyring initialized [ 2.987583] NET: Registered protocol family 38 [ 2.989845] Key type asymmetric registered [ 2.991606] Asymmetric key parser 'x509' registered [ 2.993691] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.997061] io scheduler mq-deadline registered [ 2.998822] io scheduler kyber registered [ 3.000803] io scheduler bfq registered [ 3.002599] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.005605] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.008845] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.012064] ACPI: Power Button [PWRF] [ 3.017926] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.025187] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.038811] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.047643] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.066603] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.095351] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.125018] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.129315] Non-volatile memory driver v1.3 [ 3.130884] Linux agpgart interface v0.103 [ 3.164319] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.166661] vda: detected capacity change from 0 to 74698752 [ 3.184857] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.188180] vdb: detected capacity change from 0 to 1073741824 [ 3.202782] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.205687] vdc: detected capacity change from 0 to 2621440000 [ 3.220971] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.224000] vdd: detected capacity change from 0 to 2621440000 [ 3.239736] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.242566] vde: detected capacity change from 0 to 4294967296 [ 3.259202] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.262356] vdf: detected capacity change from 0 to 4294967296 [ 3.283542] libphy: Fixed MDIO Bus: probed [ 3.289662] usbcore: registered new interface driver usbserial_generic [ 3.292070] usbserial: USB Serial support registered for generic [ 3.294248] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.298880] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.300678] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.302931] mousedev: PS/2 mouse device common for all mice [ 3.306086] rtc_cmos 00:05: RTC can wake from S4 [ 3.308462] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.309760] rtc_cmos 00:05: registered as rtc0 [ 3.313632] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.315512] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.316871] intel_pstate: CPU model not supported [ 3.320654] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.324445] hid: raw HID events driver (C) Jiri Kosina [ 3.325942] usbcore: registered new interface driver usbhid [ 3.327793] usbhid: USB HID core driver [ 3.328866] drop_monitor: Initializing network drop monitor service [ 3.330559] Initializing XFRM netlink socket [ 3.332139] NET: Registered protocol family 10 [ 3.335289] Segment Routing with IPv6 [ 3.336731] NET: Registered protocol family 17 [ 3.338793] mpls_gso: MPLS GSO support [ 3.344959] RAS: Correctable Errors collector initialized. [ 3.347152] AVX version of gcm_enc/dec engaged. [ 3.348776] AES CTR mode by8 optimization enabled [ 3.425077] sched_clock: Marking stable (3425019923, 0)->(4345818635, -920798712) [ 3.428722] registered taskstats version 1 [ 3.430461] Loading compiled-in X.509 certificates [ 3.432204] zswap: loaded using pool lzo/zbud [ 3.455091] Key type big_key registered [ 3.468124] Key type encrypted registered [ 3.469699] ima: No TPM chip found, activating TPM-bypass! [ 3.471791] ima: Allocated hash algorithm: sha1 [ 3.473539] ima: No architecture policies found [ 3.475297] evm: Initialising EVM extended attributes: [ 3.477268] evm: security.selinux [ 3.478389] evm: security.ima [ 3.479262] evm: security.capability [ 3.480574] evm: HMAC attrs: 0x1 [ 3.483094] rtc_cmos 00:05: setting system clock to 2026-08-15 20:48:44 UTC (1786826924) [ 3.488831] debug: unmapping init [mem 0xffffffffae003000-0xffffffffae1fffff] [ 3.491752] debug: unmapping init [mem 0xffffffffacd82000-0xffffffffad058fff] [ 3.500083] Write protecting the kernel read-only data: 28672k [ 3.504723] debug: unmapping init [mem 0xffffffffab403000-0xffffffffab5fffff] [ 3.507215] debug: unmapping init [mem 0xffffffffabd14000-0xffffffffabdfffff] [ 3.541687] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.551192] systemd[1]: Detected virtualization kvm. [ 3.553559] systemd[1]: Detected architecture x86-64. [ 3.555567] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.583732] systemd[1]: No hostname configured. [ 3.585904] systemd[1]: Set hostname to . [ 3.588108] random: systemd: uninitialized urandom read (16 bytes read) [ 3.590521] systemd[1]: Initializing machine ID from random generator. [ 3.737562] random: systemd: uninitialized urandom read (16 bytes read) [ 3.740095] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.744764] random: systemd: uninitialized urandom read (16 bytes read) [ 3.746970] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.750720] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Journal Service... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.393584] device-mapper: uevent: version 1.0.3 [ 4.395223] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 5.026306] random: fast init done [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.198687] virtio_net virtio0 ens2: renamed from eth0 [ 5.248194] scsi host0: ata_piix [ 5.276756] scsi host1: ata_piix [ 5.299208] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.301562] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.991460] random: crng init done [ 9.993662] random: 7 urandom warning(s) missed due to ratelimiting [ 10.635335] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 13.137899] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.107153] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.688804] SELinux: Disabled at runtime. [ 16.794449] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.831406] systemd[1]: Detected virtualization kvm. [ 16.839187] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.828928] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.842207] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.853126] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.867516] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.876218] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.919229] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.960895] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target RPC Port Mapper. [ 19.472388] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 20.883081] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.248996] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.368271] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.859872] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.904132] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 29.122217] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (11s / no limit)[ 30.018121] hrtimer: interrupt took 5038949 ns [ 30.154116] NFS: Registering the id_resolver key type [ 30.165122] Key type id_resolver registered [ 30.168233] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg118-server login: [ 103.199750] libcfs: loading out-of-tree module taints kernel. [ 103.305510] Key type ._llcrypt registered [ 103.308665] Key type .llcrypt registered [ 103.429953] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 120.930091] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 123.363407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 123.417318] alg: No test for adler32 (adler32-zlib) [ 125.414846] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 126.422334] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 128.224164] Key type lgssc registered [ 130.285149] Lustre: Echo OBD driver; http://www.lustre.org/ [ 154.429737] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 202.362636] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 216.821907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 216.848756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 218.185908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 218.222906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 218.385312] Lustre: lustre-MDT0000: new disk, initializing [ 218.523399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 218.587984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 222.890077] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.106626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 239.266889] Lustre: 6526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 239.306606] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 239.313407] Lustre: Skipped 1 previous similar message [ 239.418176] Lustre: lustre-MDT0001: new disk, initializing [ 239.533042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 239.580971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 239.585974] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 245.754418] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 251.489498] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 262.828365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 263.041122] Lustre: lustre-OST0000: new disk, initializing [ 263.045905] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 263.052850] Lustre: 8465:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 263.106815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 269.627626] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 270.361624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 270.380020] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 270.495544] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 284.874392] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 285.059679] Lustre: lustre-OST0001: new disk, initializing [ 285.069559] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 285.084954] Lustre: 9539:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 285.185894] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 292.174273] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 293.441935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 293.450581] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 293.598645] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 306.412643] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 315.249917] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 322.414744] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing check_logdir /tmp/testlogs/ [ 328.491815] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing yml_node [ 333.482598] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 336.214953] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 339.041345] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 341.039977] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 15 16:54:21 EDT 2026 [ 361.921209] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 363.835869] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 380.899376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 380.902737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 380.913863] Lustre: Skipped 1 previous similar message [ 380.940611] Lustre: Skipped 3 previous similar messages [ 386.021523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 386.035726] Lustre: Skipped 3 previous similar messages [ 386.967875] Lustre: server umount lustre-MDT0000 complete [ 391.064810] LustreError: 6517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827312 with bad export cookie 9042332134355319384 [ 391.066260] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 391.078256] LustreError: 6517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 391.151782] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 391.169915] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 391.179725] Lustre: Skipped 3 previous similar messages [ 391.202902] Lustre: Skipped 1 previous similar message [ 396.257296] LustreError: 8751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 396.257752] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 396.273112] LustreError: 8751:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 396.291930] Lustre: Skipped 1 previous similar message [ 397.626457] Lustre: server umount lustre-MDT0001 complete [ 402.207261] Lustre: server umount lustre-OST0000 complete [ 406.844303] Lustre: server umount lustre-OST0001 complete [ 414.272074] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 426.125526] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 473.825360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 474.220897] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 474.287481] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 474.453873] Lustre: lustre-MDT0000: new disk, initializing [ 474.602736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 474.638368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 482.334776] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 497.528034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 497.690506] Lustre: 15093:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 497.757232] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 497.772218] Lustre: Skipped 1 previous similar message [ 497.865243] Lustre: lustre-MDT0001: new disk, initializing [ 498.096551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 498.178085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 498.188602] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 503.932358] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 515.578665] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 523.157491] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 531.285681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 531.567449] Lustre: lustre-OST0000: new disk, initializing [ 531.570347] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 531.577269] Lustre: 17067:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 531.654576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 533.342559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 533.358453] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 533.631751] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 539.575424] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 551.977854] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 558.252230] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 558.603817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 564.198282] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 564.510216] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 569.315812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 569.316977] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 569.349067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 569.365758] Lustre: Skipped 1 previous similar message [ 573.486860] Lustre: server umount lustre-OST0000 complete [ 579.556051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.563720] Lustre: Skipped 2 previous similar messages [ 579.563928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 579.571992] Lustre: Skipped 1 previous similar message [ 592.353729] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 592.885531] Lustre: server umount lustre-MDT0000 complete [ 594.925394] LustreError: 17015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.949031] LustreError: 17015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 597.620729] LustreError: 15890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827518 with bad export cookie 9042332134355320917 [ 597.628278] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 597.648261] LustreError: 15890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 598.507722] Lustre: server umount lustre-MDT0001 complete [ 606.176056] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 16:58:46 (1786827526) [ 624.965833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 625.753161] LustreError: 19944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 625.845786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 631.129444] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 631.268449] LustreError: 19945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 635.363619] LustreError: 19944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 640.145655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 646.150826] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 656.282592] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 663.311428] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 665.755515] Lustre: Failing over lustre-MDT0000 [ 666.089493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 666.098592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 666.117455] LustreError: 19945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 668.163813] Lustre: server umount lustre-MDT0000 complete [ 685.539464] LustreError: 19944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 685.573759] LustreError: 19944:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 686.561179] Lustre: 3659:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786827591/real 1786827591] req@ffff92a70ef0d180 x1873623957801088/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786827607 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 686.596083] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 688.305898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 696.807675] LustreError: 3655:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92a70f3d7100 x1873623957803520/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 697.115799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 697.127885] Lustre: Skipped 1 previous similar message [ 697.189968] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 702.147158] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 702.437535] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 702.444839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 702.468395] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 711.707740] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 717.795298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 717.807601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 717.822636] Lustre: Skipped 1 previous similar message [ 717.830828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 717.836494] Lustre: Skipped 4 previous similar messages [ 723.239707] Lustre: server umount lustre-MDT0000 complete [ 725.985437] LustreError: 20672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 725.993445] LustreError: 20672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 727.860471] LustreError: 19925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827648 with bad export cookie 9042332134355322338 [ 727.869052] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 727.882436] LustreError: 19925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 728.186552] Lustre: server umount lustre-MDT0001 complete [ 738.263375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 747.056457] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 758.773462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 764.834913] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 773.851770] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 781.928990] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 783.683400] Lustre: Failing over lustre-MDT0000 [ 784.283498] Lustre: server umount lustre-MDT0000 complete [ 784.740724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 784.758198] Lustre: Skipped 2 previous similar messages [ 789.988707] LustreError: 24028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 790.016150] LustreError: 24028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 801.256543] Lustre: 3657:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786827705/real 1786827705] req@ffff92a603603100 x1873623957831168/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786827721 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 801.274554] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 805.087692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 812.517740] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7d7cd12af959498c [ 812.545472] Lustre: MGC192.168.201.118@tcp: Connection restored to 0@lo (at 0@lo) [ 812.556025] Lustre: Skipped 1 previous similar message [ 813.037583] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 818.189353] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 818.206941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 818.262369] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 818.613985] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 831.264758] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 835.726602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 835.740468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 835.748075] Lustre: Skipped 1 previous similar message [ 835.762994] Lustre: Skipped 3 previous similar messages [ 842.272531] Lustre: server umount lustre-MDT0000 complete [ 846.488507] LustreError: 23268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827767 with bad export cookie 9042332134355323276 [ 846.502334] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 846.512257] LustreError: 23268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 847.027569] Lustre: server umount lustre-MDT0001 complete [ 856.155788] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 17:02:56 (1786827776) [ 874.227240] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 885.492455] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 932.968060] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 944.468040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 944.659522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 944.685074] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 944.763645] Lustre: lustre-MDT0000: new disk, initializing [ 944.829859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 944.835687] Lustre: Skipped 3 previous similar messages [ 944.854569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 949.807430] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 961.646923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 961.804249] Lustre: 29681:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 962.070897] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 966.559090] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 977.076486] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 983.802667] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 990.763589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 991.075408] Lustre: 31653:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 992.652468] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 992.699236] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 992.811239] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 992.844031] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 997.836603] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1008.448994] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1015.437641] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1043.941118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1043.946756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1043.958154] LustreError: Skipped 1 previous similar message [ 1043.973817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1043.987740] Lustre: Skipped 3 previous similar messages [ 1045.945259] Lustre: server umount lustre-OST0000 complete [ 1054.181294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1057.506143] Lustre: server umount lustre-MDT0000 complete [ 1059.299168] LustreError: 29693:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1059.351475] LustreError: 29693:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 1061.669464] LustreError: 29675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827982 with bad export cookie 9042332134355323682 [ 1061.676647] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1061.680240] LustreError: 29675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1062.195380] Lustre: server umount lustre-MDT0001 complete [ 1074.269658] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1077.207848] Key type lgssc unregistered [ 1077.608760] LNet: 33635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1077.616096] LNetError: 33635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1078.637628] LNet: Removed LNI 192.168.201.118@tcp [ 1079.977337] Key type .llcrypt unregistered [ 1079.988724] Key type ._llcrypt unregistered [ 1108.552527] Key type ._llcrypt registered [ 1108.553994] Key type .llcrypt registered [ 1108.735809] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1110.197709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1110.222947] alg: No test for adler32 (adler32-zlib) [ 1111.259336] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 1111.543813] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1113.248306] Key type lgssc registered [ 1114.476615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1124.591294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1124.647855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1126.234523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1132.605750] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1136.612355] LustreError: 35050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1136.645378] LustreError: 35050:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1141.731119] LustreError: 35049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1142.006185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1142.829665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1149.118745] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1157.173413] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1165.559288] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1174.833953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1175.390812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1184.009996] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1188.760373] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1193.079402] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1202.183700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1211.912169] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 17:08:52 (1786828132) [ 1214.390405] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1216.334517] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1223.547585] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 17:09:03 (1786828143) [ 1233.304864] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 17:09:13 (1786828153) [ 1234.504602] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1243.765994] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1243.771736] Lustre: Skipped 2 previous similar messages [ 1245.693674] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 17:09:26 (1786828166) [ 1247.864522] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1247.876526] Lustre: Skipped 2 previous similar messages [ 1256.031459] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1256.037160] Lustre: Skipped 4 previous similar messages [ 1269.166722] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 17:09:49 (1786828189) [ 1271.861476] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1275.564534] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1284.933419] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 17:10:05 (1786828205) [ 1287.152693] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1287.160748] Lustre: Skipped 6 previous similar messages [ 1303.707940] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 17:10:24 (1786828224) [ 1319.918244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1319.918847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1319.936254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1319.948807] Lustre: Skipped 1 previous similar message [ 1325.029756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1325.040039] Lustre: Skipped 1 previous similar message [ 1330.148928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1330.151090] Lustre: Skipped 1 previous similar message [ 1332.197972] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1332.393104] Lustre: server umount lustre-OST0000 complete [ 1337.313694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1337.324268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1337.343483] Lustre: Skipped 1 previous similar message [ 1337.348177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1340.390458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1342.871987] Lustre: server umount lustre-MDT0000 complete [ 1345.505930] LustreError: 35049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1345.516564] LustreError: 35049:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1346.732549] LustreError: 35029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828267 with bad export cookie 15217542673437314176 [ 1346.737172] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1346.747093] LustreError: 35029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1347.063962] Lustre: server umount lustre-MDT0001 complete [ 1359.241892] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1361.999519] Key type lgssc unregistered [ 1362.363732] LNet: 42904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1362.369725] LNetError: 42904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1362.384565] LNet: Removed LNI 192.168.201.118@tcp [ 1363.158474] Key type .llcrypt unregistered [ 1363.160316] Key type ._llcrypt unregistered [ 1382.361306] Key type ._llcrypt registered [ 1382.364366] Key type .llcrypt registered [ 1382.464690] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 1399.065968] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1400.502527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1400.639317] alg: No test for adler32 (adler32-zlib) [ 1401.711489] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 1402.108768] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1403.953591] Key type lgssc registered [ 1405.269020] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1454.277712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1454.314591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1455.884038] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1456.001811] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1456.154995] Lustre: lustre-MDT0000: new disk, initializing [ 1456.342305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1456.377128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1461.671574] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1474.125879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1474.294181] Lustre: 46060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1474.384935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1474.389314] Lustre: Skipped 1 previous similar message [ 1474.551254] Lustre: lustre-MDT0001: new disk, initializing [ 1474.712703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1474.743872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1474.755627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1480.401716] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1492.406390] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1498.854618] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1506.945212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1507.417362] Lustre: lustre-OST0000: new disk, initializing [ 1507.420953] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1507.431964] Lustre: 48030:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1507.610243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1514.767529] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1516.658458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1516.671173] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1516.770141] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1525.374277] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1531.566459] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1531.831108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1537.271097] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1537.499702] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1542.632229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1542.640776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1542.647827] Lustre: Skipped 1 previous similar message [ 1542.673222] Lustre: Skipped 1 previous similar message [ 1545.668986] Lustre: server umount lustre-OST0000 complete [ 1551.841908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1551.858657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1551.883176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1552.869569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1552.891813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1552.895844] Lustre: Skipped 1 previous similar message [ 1556.489488] Lustre: server umount lustre-MDT0000 complete [ 1560.362984] LustreError: 46054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828481 with bad export cookie 2641879362768116514 [ 1560.364724] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1560.373202] LustreError: 46054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1560.744268] Lustre: server umount lustre-MDT0001 complete [ 1570.576757] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 17:14:51 (1786828491) [ 1577.100543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1577.629744] LustreError: 50490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1577.659624] LustreError: 50490:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1577.728728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1582.407693] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1583.073462] LustreError: 50491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1587.175450] LustreError: 50490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1591.353491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1591.981719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1597.486110] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1608.006631] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1614.877485] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1622.785324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1623.434807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1631.379382] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1640.905469] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1656.496233] LustreError: 53039:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1657.994842] Lustre: Modifying parameter general.timeout=1 in log params [ 1659.337802] Lustre: Modifying parameter general.timeout=2 in log params [ 1662.486140] Lustre: Modifying parameter general.timeout=5 in log params [ 1662.491727] Lustre: Skipped 2 previous similar messages [ 1666.674205] Lustre: Modifying parameter general.timeout=9 in log params [ 1666.677449] Lustre: Skipped 3 previous similar messages [ 1675.227129] Lustre: Modifying parameter general.timeout=17 in log params [ 1675.232463] Lustre: Skipped 7 previous similar messages [ 1691.266510] Lustre: Modifying parameter general.timeout=33 in log params [ 1691.275638] Lustre: Skipped 15 previous similar messages [ 1723.739931] Lustre: Modifying parameter general.timeout=63 in log params [ 1723.749292] Lustre: Skipped 29 previous similar messages [ 1796.105588] Lustre: Disabling parameter general.timeout= in log params [ 1796.107537] Lustre: Skipped 66 previous similar messages [ 1799.699707] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 17:18:40 (1786828720) [ 1803.414416] Lustre: 59999:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1805.622504] Lustre: 60097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1807.787946] Lustre: 60196:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1813.754491] Lustre: 60507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1813.766591] Lustre: 60507:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1846.318627] Lustre: 61869:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1846.365380] LustreError: 61869:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1852.247656] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 17:19:32 (1786828772) [ 1868.258442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1868.275145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1868.298152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1873.613304] Lustre: server umount lustre-MDT0000 complete [ 1874.408493] LustreError: 53672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1874.447972] LustreError: 53672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1877.395916] LustreError: 50472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828798 with bad export cookie 2641879362768117396 [ 1877.406218] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1877.410614] LustreError: 50472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1877.791849] Lustre: server umount lustre-MDT0001 complete [ 1892.934466] Lustre: server umount lustre-OST0000 complete [ 1924.250720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1924.681619] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1924.783136] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1924.799659] Lustre: Skipped 22 previous similar messages [ 1925.143849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1930.619565] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1943.718563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1943.895962] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1943.942495] Lustre: 64403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1950.785198] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1959.868696] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1966.605296] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1973.840241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1974.134090] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1974.497100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1974.508344] Lustre: Skipped 1 previous similar message [ 1981.012629] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1988.945969] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1991.067944] Lustre: 66971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 2002.402239] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2002.415558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2002.429498] Lustre: Skipped 2 previous similar messages [ 2002.438359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2002.443436] Lustre: Skipped 3 previous similar messages [ 2005.758810] Lustre: server umount lustre-OST0000 complete [ 2011.106783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2011.109958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.126218] Lustre: Skipped 1 previous similar message [ 2011.146072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2011.148981] Lustre: Skipped 1 previous similar message [ 2017.623570] Lustre: server umount lustre-MDT0000 complete [ 2018.275543] LustreError: 66159:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2018.294220] LustreError: 66159:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2022.363193] LustreError: 64395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828943 with bad export cookie 2641879362768120504 [ 2022.377943] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2022.388088] LustreError: 64395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2022.884892] Lustre: server umount lustre-MDT0001 complete [ 2040.497303] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2044.220958] Key type lgssc unregistered [ 2044.713903] LNet: 68198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2044.725742] LNetError: 68198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2045.803851] LNet: Removed LNI 192.168.201.118@tcp [ 2046.823055] Key type .llcrypt unregistered [ 2046.825098] Key type ._llcrypt unregistered [ 2078.839696] Key type ._llcrypt registered [ 2078.842483] Key type .llcrypt registered [ 2078.998982] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2080.418388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2080.520878] alg: No test for adler32 (adler32-zlib) [ 2081.590750] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 2081.945113] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2083.664195] Key type lgssc registered [ 2084.839873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2093.763384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2093.798640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2095.291755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2099.813911] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2105.831275] LustreError: 69614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2105.847602] LustreError: 69614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2109.215049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2109.920681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2114.810508] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2122.809691] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2128.936684] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2137.166947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2137.948136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2145.812206] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2153.946529] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2162.391269] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2171.363976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2171.381573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2171.393694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2174.951227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2174.954463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2174.979206] Lustre: Skipped 2 previous similar messages [ 2176.489634] Lustre: server umount lustre-MDT0000 complete [ 2180.066732] LustreError: 69608:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2180.102241] LustreError: 69608:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2180.559512] LustreError: 69593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829101 with bad export cookie 633035799522456876 [ 2180.562177] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2180.569961] LustreError: 69593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2180.873831] Lustre: server umount lustre-MDT0001 complete [ 2195.787385] Lustre: server umount lustre-OST0000 complete [ 2224.116164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2224.549116] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2224.573133] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2224.920335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2229.918745] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2239.439633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2239.547903] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2239.587318] Lustre: 74492:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2240.233299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2245.915331] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2254.912213] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2261.608564] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2268.576343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2268.741457] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2269.090290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2276.308519] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2284.923410] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2286.789603] Lustre: 77068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2288.834293] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2298.854434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2298.855153] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2298.858832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2298.870096] Lustre: Skipped 2 previous similar messages [ 2298.908318] LustreError: Skipped 1 previous similar message [ 2300.212384] Lustre: server umount lustre-OST0000 complete [ 2306.529809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2306.541519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2306.556720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2306.564750] Lustre: Skipped 1 previous similar message [ 2310.706709] Lustre: server umount lustre-MDT0000 complete [ 2314.209915] LustreError: 74501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2314.245924] LustreError: 74501:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2315.185674] LustreError: 74484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829236 with bad export cookie 633035799522457723 [ 2315.204935] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2315.212025] LustreError: 74484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2315.706771] Lustre: server umount lustre-MDT0001 complete [ 2328.273226] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2331.194803] Key type lgssc unregistered [ 2331.516180] LNet: 78282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2331.523852] LNetError: 78282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2331.546859] LNet: Removed LNI 192.168.201.118@tcp [ 2332.371451] Key type .llcrypt unregistered [ 2332.373449] Key type ._llcrypt unregistered [ 2359.853853] Key type ._llcrypt registered [ 2359.857793] Key type .llcrypt registered [ 2359.951163] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2360.915596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2360.935787] alg: No test for adler32 (adler32-zlib) [ 2362.054937] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 2362.284419] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2363.952286] Key type lgssc registered [ 2365.345629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2373.682334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2373.696050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2375.243486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2380.087227] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2385.379531] LustreError: 79697:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2385.412549] LustreError: 79697:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2387.978279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2388.994324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2394.383662] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2401.724258] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2407.021156] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2414.423644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2415.157114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2423.137720] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2430.169798] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2440.705876] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2441.511839] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2443.450123] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2443.460754] Lustre: Skipped 1 previous similar message [ 2446.368826] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2446.370766] Lustre: Skipped 2 previous similar messages [ 2451.517204] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2451.529253] Lustre: Skipped 4 previous similar messages [ 2459.739526] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2459.742031] Lustre: Skipped 6 previous similar messages [ 2476.141147] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2476.145985] Lustre: Skipped 16 previous similar messages [ 2509.252940] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2509.256158] Lustre: Skipped 32 previous similar messages [ 2547.321156] Lustre: 87802:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2550.212465] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 17:31:10 (1786829470) [ 2551.716422] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2553.250959] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 17:31:14 (1786829474) [ 2558.443840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2558.453869] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2558.468097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2562.026633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2562.029781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2562.053613] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2563.557628] Lustre: server umount lustre-OST0000 complete [ 2567.164531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2567.172472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2567.192028] Lustre: Skipped 1 previous similar message [ 2567.207401] Lustre: Skipped 2 previous similar messages [ 2572.257797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2572.264517] Lustre: Skipped 1 previous similar message [ 2573.337920] Lustre: server umount lustre-MDT0000 complete [ 2576.783069] LustreError: 79677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829497 with bad export cookie 17381754733174376851 [ 2576.791912] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2577.170209] Lustre: server umount lustre-MDT0001 complete [ 2587.013768] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2589.568395] Key type lgssc unregistered [ 2589.911891] LNet: 89053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2589.915609] LNetError: 89053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2589.926251] LNet: Removed LNI 192.168.201.118@tcp [ 2590.794511] Key type .llcrypt unregistered [ 2590.799505] Key type ._llcrypt unregistered [ 2594.955311] Key type ._llcrypt registered [ 2594.958267] Key type .llcrypt registered [ 2595.056354] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2599.781812] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules [ 2600.881334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2601.039349] alg: No test for adler32 (adler32-zlib) [ 2602.169285] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 2602.427781] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2604.153292] Key type lgssc registered [ 2605.731959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2606.929619] LustreError: 90256:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2624.903634] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2628.752887] LustreError: 90256:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2635.737402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2635.757149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2637.343220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2642.330705] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2647.541978] LustreError: 91281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2647.560074] LustreError: 91281:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2652.645631] LustreError: 91282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2653.995731] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 17:32:53 (1786829573) [ 2657.785353] LustreError: 91281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2659.843969] LustreError: 91314:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2659.867718] LustreError: 91314:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2659.889613] LustreError: 91314:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 2660.251444] Lustre: server umount lustre-MDT0000 complete [ 2674.108888] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2677.857174] Key type lgssc unregistered [ 2678.351553] LNet: 92809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2678.356169] LNetError: 92809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2678.371268] LNet: Removed LNI 192.168.201.118@tcp [ 2679.801411] Key type .llcrypt unregistered [ 2679.804216] Key type ._llcrypt unregistered [ 2701.845736] Key type ._llcrypt registered [ 2701.849760] Key type .llcrypt registered [ 2702.000264] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2703.621691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2703.747245] alg: No test for adler32 (adler32-zlib) [ 2704.822304] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 2705.116179] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2706.840344] Key type lgssc registered [ 2708.283750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2716.235725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2716.247227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2717.736888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2722.154673] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2727.910650] LustreError: 94029:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2727.934112] LustreError: 94029:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2733.033361] LustreError: 94030:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2733.321788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2734.158670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2738.875838] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2746.994773] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2753.422822] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2760.232127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2760.918696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2767.134265] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2776.004390] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2829.310387] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 17:35:49 (1786829749) [ 2831.135890] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2833.244159] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 17:35:53 (1786829753) [ 2840.033943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2840.037379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2840.057503] Lustre: Skipped 1 previous similar message [ 2840.068438] Lustre: Skipped 2 previous similar messages [ 2842.993248] Lustre: server umount lustre-MDT0000 complete [ 2845.158686] LustreError: 95035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2845.175193] LustreError: 95035:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2845.789158] LustreError: 94010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829766 with bad export cookie 14872246436651848030 [ 2845.799829] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2846.124685] Lustre: server umount lustre-MDT0001 complete [ 2860.200581] Lustre: server umount lustre-OST0000 complete [ 2869.437467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2869.871480] LustreError: 98333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2870.024928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2874.141237] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2875.370966] LustreError: 98333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2875.376272] LustreError: 98333:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2881.828569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2882.842701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2888.141917] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2897.341773] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2903.420647] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2915.647027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2915.876545] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2915.886307] LustreError: 98321:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2915.904778] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2915.915481] LustreError: 100416:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2915.923716] LustreError: 100416:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2915.928618] LustreError: 100416:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2916.023397] Lustre: server umount lustre-OST0000 complete [ 2916.032860] LustreError: 100416:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2923.997835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2924.128845] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2924.135203] LustreError: 98321:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2924.145622] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2924.158975] LustreError: 100877:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2924.163789] LustreError: 100877:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2924.174204] LustreError: 100877:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2924.244711] Lustre: server umount lustre-OST0000 complete [ 2924.250159] LustreError: 100877:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2932.907669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2933.084827] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2933.088130] Lustre: Found index 0 for lustre-OST0000, updating log [ 2933.093090] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2933.183563] Lustre: lustre-OST0000: new disk, initializing [ 2933.187254] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2933.195271] Lustre: 101412:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2933.517464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2939.907819] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2943.920696] Lustre: Failing over lustre-OST0000 [ 2944.076170] Lustre: server umount lustre-OST0000 complete [ 2948.577350] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2948.593384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2948.623551] Lustre: Skipped 1 previous similar message [ 2948.639941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2950.115896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2950.116755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2950.150383] Lustre: Skipped 1 previous similar message [ 2952.226343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2952.241431] Lustre: Skipped 1 previous similar message [ 2957.284657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2957.306334] Lustre: Skipped 1 previous similar message [ 2958.452439] Lustre: server umount lustre-MDT0000 complete [ 2962.405609] LustreError: 98330:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2962.440121] LustreError: 98330:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2963.396052] LustreError: 98313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829884 with bad export cookie 14872246436651850767 [ 2963.398818] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2963.410471] LustreError: 98313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2963.772314] Lustre: server umount lustre-MDT0001 complete [ 2973.373325] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 17:38:13 (1786829893) [ 2993.678646] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3007.317988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3007.881983] LustreError: 103760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3008.073948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3014.350739] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3026.529423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3032.387301] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3035.638696] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3042.949702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3043.622850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3043.634270] Lustre: Skipped 1 previous similar message [ 3043.645626] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3049.018258] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3051.818867] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3051.824185] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 3051.842721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 3052.846557] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:08 [ 3052.868636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 3052.903499] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3052.903708] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3052.909934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 3056.195139] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3056.303934] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3056.421373] Lustre: lustre-OST0001: new disk, initializing [ 3056.425146] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3056.436513] Lustre: 106082:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3062.219844] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3063.373451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3063.382536] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3063.479931] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3070.136789] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3080.707396] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3085.117973] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.18@tcp (stopping) [ 3088.866308] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3088.881724] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3090.882622] Lustre: server umount lustre-MDT0001 complete [ 3094.506199] LustreError: 103761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3094.529519] LustreError: 103761:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3099.643105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3099.782550] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3099.794307] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3099.882520] Lustre: 103749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3101.819190] LustreError: 107659:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3101.827784] LustreError: 107659:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3101.837192] LustreError: 103752:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3101.934617] LustreError: 107670:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.118@tcp: cfg command failed: rc = -17 [ 3101.943733] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3101.943733] [ 3104.730691] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3105.798703] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3113.971274] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 17:40:34 (1786830034) [ 3131.812992] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3148.404948] Lustre: 109695:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3172.013222] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3176.041986] Lustre: 110831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3269.089361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3269.096452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3269.112228] Lustre: Skipped 2 previous similar messages [ 3269.122573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3269.129711] Lustre: Skipped 5 previous similar messages [ 3273.492235] Lustre: server umount lustre-MDT0000 complete [ 3274.722111] LustreError: 103757:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3274.733750] LustreError: 103757:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3276.992201] LustreError: 103741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830198 with bad export cookie 14872246436651851705 [ 3277.001038] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3277.009745] LustreError: 103741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3277.228579] Lustre: server umount lustre-MDT0001 complete [ 3291.984400] Lustre: server umount lustre-OST0000 complete [ 3295.200110] Lustre: 93341:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786830200/real 1786830200] req@ffff92a73909a680 x1873626662271488/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786830216 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3295.238092] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3295.249432] Lustre: Skipped 3 previous similar messages [ 3297.456109] Lustre: server umount lustre-OST0001 complete [ 3304.286982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3314.710707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3324.594030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3334.852302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3359.192641] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3371.845508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3371.864790] Lustre: lustre-MDT0000: reset Object Index mappings [ 3372.644386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3372.650748] Lustre: Skipped 2 previous similar messages [ 3378.755418] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3390.908582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3390.927735] Lustre: lustre-MDT0001: reset Object Index mappings [ 3397.289888] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3401.328519] Lustre: 115933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3409.922910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3416.177377] LustreError: 116286:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3416.213243] LustreError: 116286:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3416.245190] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 3418.306176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:386 to 0x280000401:418) [ 3418.842212] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3428.155084] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3434.481523] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3434.506075] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3435.826947] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3445.395589] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3451.463554] Lustre: 117810:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3505.085362] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 17:47:05 (1786830425) [ 3531.750085] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3531.756369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3531.761175] Lustre: Skipped 2 previous similar messages [ 3531.765900] Lustre: Skipped 4 previous similar messages [ 3537.716872] Lustre: server umount lustre-MDT0000 complete [ 3541.607258] LustreError: 114763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830462 with bad export cookie 14872246436651898801 [ 3541.615793] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3541.620812] LustreError: 114763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3548.211400] Lustre: server umount lustre-MDT0001 complete [ 3552.702780] LustreError: 118574:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000400:0xba:0x0].0x0 (ffff92a73935d500) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3558.956774] Lustre: server umount lustre-OST0000 complete [ 3576.288156] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3576.689267] Lustre: server umount lustre-OST0001 complete [ 3583.103906] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 3593.343929] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3632.073139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3632.374455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3632.420512] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3632.578628] Lustre: lustre-MDT0000: new disk, initializing [ 3632.643391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3632.647225] Lustre: Skipped 3 previous similar messages [ 3632.664248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3637.162417] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3644.447066] Lustre: server umount lustre-MDT0000 complete [ 3653.786610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3654.064171] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3654.076888] Lustre: 122086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3654.086932] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3660.037343] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3665.228807] Lustre: server umount lustre-MDT0000 complete [ 3676.048444] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 17:49:55 (1786830595) [ 3677.893460] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3680.313888] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 17:50:00 (1786830600) [ 3699.500495] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3710.556464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3716.150158] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3726.179073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3726.340843] Lustre: 124136:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3726.464698] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3726.472097] Lustre: Skipped 1 previous similar message [ 3726.584914] Lustre: lustre-MDT0001: new disk, initializing [ 3726.755448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3726.764748] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3731.331529] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3735.966646] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3743.656054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3744.015097] Lustre: lustre-OST0000: new disk, initializing [ 3744.026952] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3744.047661] Lustre: 125717:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3745.773281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3745.788332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3745.932799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3751.407470] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3763.110318] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3763.253604] Lustre: lustre-OST0001: new disk, initializing [ 3763.258977] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3763.266614] Lustre: 126587:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3765.427789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3765.446199] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3765.559751] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3771.038933] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3781.528620] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3786.269615] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4182.448155] Lustre: 133002:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4182.458487] Lustre: 133002:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4182.473755] Lustre: 133002:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4182.480361] Lustre: 133002:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4182.485516] Lustre: 133002:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4182.491825] Lustre: 133002:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4468.617448] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 18:03:08 (1786831388) [ 4484.580254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4484.586774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4484.595251] Lustre: Skipped 2 previous similar messages [ 4484.599974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4484.602933] Lustre: Skipped 10 previous similar messages [ 4489.968858] Lustre: server umount lustre-MDT0000 complete [ 4492.786267] LustreError: 127422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4492.816993] LustreError: 127422:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 4493.724099] LustreError: 124128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831414 with bad export cookie 14872246436651967926 [ 4493.725149] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4493.734684] LustreError: 124128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4497.893871] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4497.896702] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4497.904981] Lustre: Skipped 4 previous similar messages [ 4497.918565] Lustre: Skipped 5 previous similar messages [ 4508.132349] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4508.567280] Lustre: server umount lustre-MDT0001 complete [ 4518.431640] Lustre: server umount lustre-OST0000 complete [ 4536.288330] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4536.460807] Lustre: server umount lustre-OST0001 complete [ 4544.366871] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 4561.473889] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4601.064123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4601.226189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4601.247634] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4601.333876] Lustre: lustre-MDT0000: new disk, initializing [ 4601.402152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4601.413763] Lustre: Skipped 5 previous similar messages [ 4601.446123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4605.851874] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4617.053110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4617.264923] Lustre: 136086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4617.363833] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4617.366432] Lustre: Skipped 1 previous similar message [ 4617.484163] Lustre: lustre-MDT0001: new disk, initializing [ 4617.657073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4617.672251] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4623.859990] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4637.664856] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4645.761556] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4653.688674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4654.015973] Lustre: lustre-OST0000: new disk, initializing [ 4654.020657] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4654.028456] Lustre: 138054:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4654.152412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4654.163400] Lustre: Skipped 1 previous similar message [ 4655.937340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4655.947056] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4656.072096] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4660.720723] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4671.140200] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4681.422799] Lustre: lustre-MDD0000: changelog on [ 4684.367825] Lustre: lustre-MDD0001: changelog on [ 6728.351289] Lustre: lustre-MDD0001: changelog off [ 6730.336472] Lustre: 136092:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6731.215189] Lustre: lustre-MDD0000: changelog off [ 6739.938914] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6739.943274] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6739.948713] LustreError: Skipped 1 previous similar message [ 6739.949560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6739.956542] Lustre: Skipped 1 previous similar message [ 6739.976584] Lustre: Skipped 7 previous similar messages [ 6745.071534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6745.084165] Lustre: Skipped 1 previous similar message [ 6750.188189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6750.201915] Lustre: Skipped 1 previous similar message [ 6752.224405] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6752.430881] Lustre: server umount lustre-OST0000 complete [ 6757.859370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6757.864882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6760.418880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6760.419885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6760.423034] Lustre: Skipped 1 previous similar message [ 6762.929345] Lustre: server umount lustre-MDT0000 complete [ 6765.540222] LustreError: 136099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6765.561757] LustreError: 136099:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6767.463914] LustreError: 136080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786833688 with bad export cookie 14872246436652783489 [ 6767.469091] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6767.472908] LustreError: 136080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6767.969902] Lustre: server umount lustre-MDT0001 complete [ 6783.195358] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 6786.622352] Key type lgssc unregistered [ 6786.912497] LNet: 147216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6786.923409] LNetError: 147216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6786.946417] LNet: Removed LNI 192.168.201.118@tcp [ 6787.972247] Key type .llcrypt unregistered [ 6787.974673] Key type ._llcrypt unregistered [ 6811.516711] Key type ._llcrypt registered [ 6811.519101] Key type .llcrypt registered [ 6811.621216] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 6830.182326] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6831.810943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6831.864165] alg: No test for adler32 (adler32-zlib) [ 6832.981795] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 6833.213045] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 6834.944193] Key type lgssc registered [ 6836.289720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6878.431649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6878.469813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6879.692510] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6879.715811] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6879.782841] Lustre: lustre-MDT0000: new disk, initializing [ 6879.844725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6879.856651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6884.419794] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6898.296289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6898.446995] Lustre: 150421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6898.550555] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6898.553407] Lustre: Skipped 1 previous similar message [ 6898.719320] Lustre: lustre-MDT0001: new disk, initializing [ 6898.865715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6898.932886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6898.948982] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6903.483801] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6913.833449] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6920.468142] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6928.593366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6928.883300] Lustre: lustre-OST0000: new disk, initializing [ 6928.888355] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6928.893967] Lustre: 152393:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6928.983334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6934.696879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6934.717788] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6934.890904] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6936.260300] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6946.594362] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6953.538512] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_module obdecho/obdecho [ 6955.869151] LustreError: 153397:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6955.883898] LustreError: 153397:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6958.369755] LustreError: 153493:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6975.464203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6975.469319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6975.484904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6975.972412] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6975.995487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6976.001118] Lustre: Skipped 1 previous similar message [ 6977.539072] Lustre: server umount lustre-MDT0000 complete [ 6981.400948] LustreError: 152391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786833902 with bad export cookie 4457206510981572517 [ 6981.402823] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6981.410105] LustreError: 152391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6981.855378] Lustre: server umount lustre-MDT0001 complete [ 6995.362455] Lustre: server umount lustre-OST0000 complete [ 7015.619725] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 7024.577804] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7058.921756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7059.132810] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7059.163288] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7059.227261] Lustre: lustre-MDT0000: new disk, initializing [ 7059.271024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7059.284160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7065.660696] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7076.338310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7076.445505] Lustre: 157208:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7076.491966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7076.495961] Lustre: Skipped 1 previous similar message [ 7076.588342] Lustre: lustre-MDT0001: new disk, initializing [ 7076.721758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7076.753901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7076.766151] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7082.058858] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7092.346606] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7098.656240] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7106.619119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7107.039689] Lustre: lustre-OST0000: new disk, initializing [ 7107.047373] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7107.057294] Lustre: 159175:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7107.215983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7113.824500] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7117.405201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7117.429845] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7117.609290] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7124.165688] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7128.738541] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7129.132939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7133.796234] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7133.991744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7138.291981] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7138.304943] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7138.316784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7138.343642] Lustre: Skipped 2 previous similar messages [ 7142.528498] Lustre: server umount lustre-OST0000 complete [ 7148.515354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7148.529202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7148.556566] Lustre: Skipped 1 previous similar message [ 7148.588824] Lustre: Skipped 2 previous similar messages [ 7153.384477] Lustre: server umount lustre-MDT0000 complete [ 7153.635269] LustreError: 157219:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7153.671434] LustreError: 157219:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7157.276590] LustreError: 157201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834078 with bad export cookie 4457206510981573476 [ 7157.279128] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7157.292275] LustreError: 157201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7157.627114] Lustre: server umount lustre-MDT0001 complete [ 7161.418776] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 18:48:01 (1786834081) [ 7163.377933] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7165.233379] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 18:48:05 (1786834085) [ 7174.294849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7174.931021] LustreError: 161589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7175.033228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7180.260556] LustreError: 161590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7180.288794] LustreError: 161590:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7180.333849] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7184.355237] LustreError: 161589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7189.480638] LustreError: 161590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7189.682846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7195.479164] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7203.205390] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7208.667036] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7217.189269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7217.755060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7217.765344] Lustre: Skipped 1 previous similar message [ 7224.923977] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7234.391076] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7245.582129] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.18@tcp (stopping) [ 7245.795854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7245.809797] Lustre: Skipped 2 previous similar messages [ 7247.120500] Lustre: server umount lustre-MDT0000 complete [ 7250.136545] LustreError: 161571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834171 with bad export cookie 4457206510981574358 [ 7250.142667] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7250.151590] LustreError: 161571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7250.682395] LustreError: 163975:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7255.804648] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.18@tcp (stopping) [ 7255.820895] Lustre: Skipped 5 previous similar messages [ 7256.458854] Lustre: server umount lustre-MDT0001 complete [ 7263.620475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7277.111759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7277.348214] LustreError: 165432:0:(llog.c:1655:llog_backup()) MGC192.168.201.118@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7277.358859] Lustre: 165432:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.118@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7280.609651] LustreError: 165437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7280.624498] LustreError: 165437:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7316.219513] LustreError: 165439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7316.239353] LustreError: 165439:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [ 7337.954321] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7337.964726] Lustre: 165432:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.118@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7378.940920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7381.755059] LustreError: 165469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7381.770805] LustreError: 165469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 44 previous similar messages [ 7383.788252] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7384.059712] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7391.944643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7400.214952] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7400.938943] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7406.832405] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7408.449126] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7415.114961] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7416.783477] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7432.537828] Lustre: server umount lustre-MDT0000 complete [ 7436.515620] LustreError: 161571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834357 with bad export cookie 4457206510981580812 [ 7436.538467] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7436.541033] LustreError: 161571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7436.782174] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7436.800081] Lustre: Skipped 1 previous similar message [ 7436.987335] Lustre: server umount lustre-MDT0001 complete [ 7440.761613] Lustre: server umount lustre-OST0000 complete [ 7461.528150] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 7470.823707] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7510.326992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7510.583976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7510.608120] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7510.715666] Lustre: lustre-MDT0000: new disk, initializing [ 7510.796327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7510.801487] Lustre: Skipped 1 previous similar message [ 7510.822686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7516.059118] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7527.792572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7527.973807] Lustre: 170534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7528.275525] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7533.153788] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7543.473770] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7549.290928] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7556.647823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7556.926356] Lustre: lustre-OST0000: new disk, initializing [ 7556.937557] Lustre: Skipped 1 previous similar message [ 7556.944521] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7556.951319] Lustre: Skipped 2 previous similar messages [ 7556.970625] Lustre: 172502:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7562.347530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7562.364420] Lustre: Skipped 1 previous similar message [ 7562.369394] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7562.494219] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7563.997153] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7574.707361] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7580.009514] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7580.322912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7585.989979] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7586.259727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7593.451855] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7593.459655] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7593.463758] LustreError: Skipped 1 previous similar message [ 7593.491981] Lustre: Skipped 1 previous similar message [ 7593.503423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7593.511367] Lustre: Skipped 2 previous similar messages [ 7594.761852] Lustre: server umount lustre-OST0000 complete [ 7600.097880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7600.103688] LustreError: Skipped 1 previous similar message [ 7605.396709] Lustre: server umount lustre-MDT0000 complete [ 7608.802406] LustreError: 170546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7608.823516] LustreError: 170546:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 7609.438524] LustreError: 170527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834530 with bad export cookie 4457206510981597647 [ 7609.439609] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7609.450105] LustreError: 170527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7609.756093] Lustre: server umount lustre-MDT0001 complete [ 7619.771490] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 18:55:40 (1786834540) [ 7627.346480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7631.781474] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7640.768157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7645.750706] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7654.750283] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7660.912450] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7669.015584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7675.639660] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7683.101856] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7695.348675] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 18:56:56 (1786834616) [ 7699.425699] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7699.430905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7699.441342] Lustre: Skipped 3 previous similar messages [ 7699.449623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7699.456111] Lustre: Skipped 3 previous similar messages [ 7705.106723] Lustre: server umount lustre-OST0000 complete [ 7712.738805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7712.746985] LustreError: Skipped 1 previous similar message [ 7715.233856] Lustre: server umount lustre-MDT0000 complete [ 7719.396449] LustreError: 174895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834640 with bad export cookie 4457206510981598529 [ 7719.406340] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7719.417985] LustreError: 174895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7719.930792] Lustre: server umount lustre-MDT0001 complete [ 7732.013480] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 7735.407171] Key type lgssc unregistered [ 7735.702897] LNet: 178646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7735.714428] LNetError: 178646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7736.746446] LNet: Removed LNI 192.168.201.118@tcp [ 7737.967522] Key type .llcrypt unregistered [ 7737.970525] Key type ._llcrypt unregistered [ 7758.394910] Key type ._llcrypt registered [ 7758.397571] Key type .llcrypt registered [ 7758.526249] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7759.738258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7759.911515] alg: No test for adler32 (adler32-zlib) [ 7761.218227] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 7761.466719] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 7763.152131] Key type lgssc registered [ 7764.522278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7772.759808] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7772.783782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7788.321364] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7793.442161] LustreError: 179891:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.118@tcp: failed processing log, type 4: rc = -110 [ 7798.561865] LustreError: 179891:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7798.585112] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7798.594989] LustreError: MGC192.168.201.118@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7798.613742] LustreError: 179891:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7798.627816] LustreError: 179891:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7798.642809] LustreError: 179891:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7798.743402] Lustre: server umount lustre-OST0000 complete [ 7798.757341] LustreError: 179891:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7808.314691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7809.806933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7814.578411] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7819.243708] LustreError: 180391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7819.280451] LustreError: 180391:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7824.213981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7824.354898] LustreError: 180390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7824.606762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7829.062751] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7837.241804] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7843.810687] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7851.323281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7851.561375] LustreError: 182206:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7851.566957] Lustre: 182206:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.118@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7851.830483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7858.816508] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7867.802226] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7870.945494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7870.950063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7870.966988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7874.529753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7874.531187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7874.551348] Lustre: Skipped 1 previous similar message [ 7876.670536] Lustre: server umount lustre-MDT0000 complete [ 7879.657159] LustreError: 180386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7879.673966] LustreError: 180386:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7880.862152] LustreError: 180372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834801 with bad export cookie 94493578850019423 [ 7880.872534] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7880.876689] LustreError: 180372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7881.326540] Lustre: server umount lustre-MDT0001 complete [ 7896.697164] Lustre: server umount lustre-OST0000 complete [ 7907.559356] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 7919.008256] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7959.866592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7960.207623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7960.241451] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7960.304923] Lustre: lustre-MDT0000: new disk, initializing [ 7960.430746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7960.446581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7966.532247] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7980.434954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7980.641078] Lustre: 185743:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7980.758417] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7980.765651] Lustre: Skipped 1 previous similar message [ 7980.873606] Lustre: lustre-MDT0001: new disk, initializing [ 7981.056853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7981.130246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7981.149679] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7987.205502] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7998.764492] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8005.361541] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8013.930847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8014.242965] Lustre: lustre-OST0000: new disk, initializing [ 8014.246620] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8014.257510] Lustre: 187714:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8014.375129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8021.622251] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8022.122132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8022.132347] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8022.288741] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8032.352159] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8037.827631] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8038.145150] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8043.635246] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8043.944796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8048.098126] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8048.111561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8048.118524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8048.137140] Lustre: Skipped 1 previous similar message [ 8048.155297] Lustre: Skipped 3 previous similar messages [ 8053.015870] Lustre: server umount lustre-OST0000 complete [ 8057.827882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8057.837571] LustreError: Skipped 1 previous similar message [ 8057.843266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8057.866751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8063.458493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8063.465797] Lustre: Skipped 3 previous similar messages [ 8064.105537] Lustre: server umount lustre-MDT0000 complete [ 8068.581786] LustreError: 185751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8068.611164] LustreError: 185751:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8069.081831] LustreError: 185736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834990 with bad export cookie 94493578850020144 [ 8069.086549] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8069.090311] LustreError: 185736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8069.951973] Lustre: server umount lustre-MDT0001 complete [ 8082.955119] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 19:03:22 (1786835002) [ 8107.225351] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 8110.762247] Key type lgssc unregistered [ 8111.067665] LNet: 190667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8111.081796] LNetError: 190667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8111.104251] LNet: Removed LNI 192.168.201.118@tcp [ 8112.209674] Key type .llcrypt unregistered [ 8112.212584] Key type ._llcrypt unregistered [ 8133.435564] Key type ._llcrypt registered [ 8133.437683] Key type .llcrypt registered [ 8133.593040] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8135.673205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8135.708692] alg: No test for adler32 (adler32-zlib) [ 8136.803253] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 8137.039282] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 8138.704207] Key type lgssc registered [ 8140.401655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8148.725474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8148.747630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8150.595385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8155.677625] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8160.743799] LustreError: 191887:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8160.756048] LustreError: 191887:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8165.079455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8165.624192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8170.610389] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8178.497431] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8184.278860] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8192.537662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8192.821876] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8192.824248] LustreError: 193751:0:(llog.c:1696:llog_backup()) MGC192.168.201.118@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8192.832420] Lustre: 193751:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.118@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8192.997876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8200.880704] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8209.753396] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8211.939975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8211.946958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8211.959762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8215.521379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8215.527079] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8215.536227] Lustre: Skipped 1 previous similar message [ 8220.646794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8220.660962] Lustre: Skipped 4 previous similar messages [ 8225.765591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8225.778429] Lustre: Skipped 2 previous similar messages [ 8226.272380] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8226.445616] Lustre: server umount lustre-MDT0000 complete [ 8230.392746] LustreError: 191868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835151 with bad export cookie 1047330381066642696 [ 8230.421633] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8230.666064] Lustre: server umount lustre-MDT0001 complete [ 8244.297971] Lustre: server umount lustre-OST0000 complete [ 8254.551190] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 8264.649894] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8300.776876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8300.963534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8300.980264] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8301.044560] Lustre: lustre-MDT0000: new disk, initializing [ 8301.102493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8301.112235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8304.954083] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8313.874116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8313.977542] Lustre: 197277:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8314.024625] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8314.027256] Lustre: Skipped 1 previous similar message [ 8314.032672] LustreError: 198059:0:(llog.c:1696:llog_backup()) MGC192.168.201.118@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8314.043022] LustreError: 198059:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8314.046490] Lustre: 198059:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.118@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8314.064062] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8314.067440] Lustre: Skipped 1 previous similar message [ 8314.126344] Lustre: lustre-MDT0001: new disk, initializing [ 8314.218629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8314.252825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8314.266077] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8318.014594] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8326.521482] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8332.470208] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8339.385625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8339.670841] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8339.674638] Lustre: Skipped 1 previous similar message [ 8339.677983] LustreError: 199220:0:(llog.c:1696:llog_backup()) MGC192.168.201.118@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8339.686053] LustreError: 199220:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8339.691199] Lustre: 199220:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.118@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8339.700616] Lustre: 199220:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8339.723755] Lustre: lustre-OST0000: new disk, initializing [ 8339.727794] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8339.733523] Lustre: 199243:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8339.837778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8345.517496] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8347.664496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8347.670655] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8347.761786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8354.399181] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8359.905496] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8360.145570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8365.244203] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8365.539168] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8368.097467] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8368.105737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8368.117183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8368.123720] Lustre: Skipped 1 previous similar message [ 8368.135718] Lustre: Skipped 1 previous similar message [ 8373.862467] Lustre: server umount lustre-OST0000 complete [ 8378.338209] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8378.342690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8378.358176] Lustre: Skipped 1 previous similar message [ 8378.374701] Lustre: Skipped 3 previous similar messages [ 8385.840096] Lustre: server umount lustre-MDT0000 complete [ 8388.579121] LustreError: 198717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8388.603555] LustreError: 198717:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8389.622874] LustreError: 197269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835310 with bad export cookie 1047330381066643410 [ 8389.641910] LustreError: 197269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8389.651835] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8389.991170] Lustre: server umount lustre-MDT0001 complete [ 8400.182267] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 19:08:40 (1786835320) [ 8417.925456] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8427.134583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8427.555654] LustreError: 202374:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8427.648448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8432.393272] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8433.131827] LustreError: 202375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8437.220899] LustreError: 202374:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8441.370823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8446.593803] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8449.802085] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8456.712445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8463.062104] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8470.910981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8471.077530] Lustre: lustre-OST0001: new disk, initializing [ 8471.080457] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8471.085616] Lustre: 204681:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8471.180462] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8471.191630] Lustre: Skipped 2 previous similar messages [ 8477.336224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8477.351357] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8477.444618] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8477.836174] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8488.754780] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8499.869610] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8502.132863] Lustre: DEBUG MARKER: ADD OST3 [ 8506.944886] loop: module loaded [ 8506.996213] loop0: detected capacity change from 0 to 204800000 [ 8507.195516] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8507.205839] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8507.218051] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8513.758605] loop0: detected capacity change from 0 to 204800000 [ 8514.190736] Lustre: DEBUG MARKER: STOP OST3 [ 8519.207615] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8519.371887] Lustre: lustre-OST0003: new disk, initializing [ 8519.383926] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8519.392231] Lustre: 206687:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8520.885943] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8521.010753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8521.022398] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8521.036452] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8522.905296] Lustre: Failing over lustre-OST0003 [ 8522.913046] LustreError: 206849:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8523.028853] Lustre: server umount lustre-OST0003 complete [ 8525.865588] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8526.139777] LustreError: 203862:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8526.320488] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8526.321439] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8534.304195] Lustre: DEBUG MARKER: START OST3 again [ 8540.482096] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8540.897474] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8540.904549] Lustre: Skipped 1 previous similar message [ 8540.918057] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8541.456442] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8542.228814] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8542.229684] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8546.760872] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8566.855405] Lustre: Failing over lustre-OST0003 [ 8567.096903] Lustre: server umount lustre-OST0003 complete [ 8567.779039] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8567.790362] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8567.792074] LustreError: Skipped 1 previous similar message [ 8567.792977] LustreError: 203860:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8567.792985] LustreError: 203860:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 8567.834926] Lustre: Skipped 2 previous similar messages [ 8585.190193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8585.205075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8585.222064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8585.225108] Lustre: Skipped 2 previous similar messages [ 8590.645876] Lustre: server umount lustre-MDT0000 complete [ 8593.756584] LustreError: 203108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835514 with bad export cookie 1047330381066644299 [ 8593.765629] LustreError: 203108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8593.767749] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8603.624108] LustreError: 202370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8603.634114] LustreError: 202370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 8608.228721] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8608.551695] Lustre: server umount lustre-MDT0001 complete [ 8612.119676] Lustre: server umount lustre-OST0000 complete [ 8615.638877] Lustre: server umount lustre-OST0001 complete [ 8622.474906] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 8629.955113] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8661.476821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8661.630730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8661.658320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8661.723374] Lustre: lustre-MDT0000: new disk, initializing [ 8661.771579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8665.900097] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8676.068658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8676.217841] Lustre: 212145:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8676.539075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8676.547711] Lustre: Skipped 1 previous similar message [ 8676.601344] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8680.581572] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8688.737690] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8693.107834] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8698.536769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8698.736621] Lustre: lustre-OST0000: new disk, initializing [ 8698.739127] Lustre: Skipped 1 previous similar message [ 8698.742599] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8698.747372] Lustre: Skipped 2 previous similar messages [ 8698.753969] Lustre: 214115:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8699.988845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8699.995243] Lustre: Skipped 1 previous similar message [ 8700.000663] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8700.087805] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8704.329810] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8712.963525] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8717.034379] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8717.228451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8721.437703] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8721.654085] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8725.986522] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8725.987073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8725.999714] LustreError: Skipped 1 previous similar message [ 8726.006617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8726.015122] Lustre: Skipped 6 previous similar messages [ 8726.036473] Lustre: Skipped 11 previous similar messages [ 8729.699688] Lustre: server umount lustre-OST0000 complete [ 8739.212845] Lustre: server umount lustre-MDT0000 complete [ 8739.298265] LustreError: 212157:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8739.307766] LustreError: 212157:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8742.752220] LustreError: 214114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835663 with bad export cookie 1047330381066646595 [ 8742.774861] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8743.078759] Lustre: server umount lustre-MDT0001 complete [ 8746.347498] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 19:14:27 (1786835667) [ 8760.390605] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 8769.312351] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8800.987077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8801.176971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8801.194617] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8801.246556] Lustre: lustre-MDT0000: new disk, initializing [ 8801.290503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8805.050949] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8814.655139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8814.806364] Lustre: 218726:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8815.087638] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8819.023554] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8827.182272] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8831.479161] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8838.201597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8838.421332] Lustre: 220693:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8840.491985] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8843.935562] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8851.895815] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8856.561763] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8856.720378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8860.532337] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8860.720698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8866.278341] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8866.281092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8866.295965] LustreError: Skipped 2 previous similar messages [ 8866.298703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8866.306898] Lustre: Skipped 3 previous similar messages [ 8866.318575] Lustre: Skipped 6 previous similar messages [ 8868.335137] Lustre: server umount lustre-OST0000 complete [ 8877.834219] LustreError: 222232:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff92a60b17d200) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 8879.937058] Lustre: server umount lustre-MDT0000 complete [ 8881.640658] LustreError: 219602:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8881.665820] LustreError: 219602:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8882.957059] LustreError: 218717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835803 with bad export cookie 1047330381066647477 [ 8882.968467] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8882.969994] LustreError: 218717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8883.243790] Lustre: server umount lustre-MDT0001 complete [ 8890.617743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8894.397829] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8901.027479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8904.807571] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8911.414723] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8915.735715] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8920.775773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8925.307756] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8931.620329] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8946.664481] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8946.678339] LustreError: Skipped 1 previous similar message [ 8952.071683] Lustre: server umount lustre-OST0000 complete [ 8961.370313] Lustre: server umount lustre-MDT0000 complete [ 8963.794530] LustreError: 222890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835884 with bad export cookie 1047330381066648359 [ 8963.804734] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8964.011113] Lustre: server umount lustre-MDT0001 complete [ 9067.875384] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 9069.973335] Key type lgssc unregistered [ 9070.194576] LNet: 226599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9070.197712] LNetError: 226599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9070.209808] LNet: Removed LNI 192.168.201.118@tcp [ 9070.917185] Key type .llcrypt unregistered [ 9070.918919] Key type ._llcrypt unregistered [ 9090.317771] Key type ._llcrypt registered [ 9090.319443] Key type .llcrypt registered [ 9090.381285] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 9103.179601] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9103.936292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9104.042406] alg: No test for adler32 (adler32-zlib) [ 9105.082823] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 9105.312131] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 9106.944740] Key type lgssc registered [ 9107.756904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9135.975995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9136.005646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9137.205654] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9137.234410] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9137.327483] Lustre: lustre-MDT0000: new disk, initializing [ 9137.413293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9137.430217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9141.182769] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9149.898671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9150.041137] Lustre: 229951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9150.100586] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9150.104299] Lustre: Skipped 1 previous similar message [ 9150.173489] Lustre: lustre-MDT0001: new disk, initializing [ 9150.268125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9150.334827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9150.346610] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9154.135759] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9161.296030] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9165.064923] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9169.897909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9170.106706] Lustre: lustre-OST0000: new disk, initializing [ 9170.110224] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9170.114966] Lustre: 231916:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9170.202327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9175.456113] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9177.642712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9177.654430] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9177.774910] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9183.160064] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9186.793399] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9186.962731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9190.708483] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9190.866087] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9192.930121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9192.939549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9192.950602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9193.447818] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9193.467866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9193.478045] Lustre: Skipped 1 previous similar message [ 9198.658696] Lustre: server umount lustre-OST0000 complete [ 9201.634531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9201.642974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9201.653373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9203.682982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9203.683886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9207.614884] Lustre: server umount lustre-MDT0000 complete [ 9208.802261] LustreError: 231112:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9208.819274] LustreError: 231112:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9210.369497] LustreError: 231915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836131 with bad export cookie 3891808705236686317 [ 9210.375216] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9210.380935] LustreError: 231915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9210.616474] Lustre: server umount lustre-MDT0001 complete [ 9223.615361] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9230.389276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9230.610969] LustreError: 234852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9230.649497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9233.892812] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9235.938170] LustreError: 234853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9240.036063] LustreError: 234852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9241.002208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9241.304723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9244.153764] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9246.410514] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9251.277051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9251.530825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9256.148617] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9261.789516] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9261.897809] Lustre: lustre-OST0001: new disk, initializing [ 9261.900589] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9261.905248] Lustre: 237158:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9265.993178] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9271.385135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9271.394917] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9271.480743] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9274.962606] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9278.010812] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9285.970354] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 19:23:27 (1786836207) [ 9287.650849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9287.661112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9287.679984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9287.682548] Lustre: Skipped 1 previous similar message [ 9293.577188] Lustre: server umount lustre-MDT0000 complete [ 9296.177859] LustreError: 235586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836217 with bad export cookie 3891808705236687199 [ 9296.181249] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9296.189741] LustreError: 235586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9296.429962] Lustre: server umount lustre-MDT0001 complete [ 9308.252385] Lustre: server umount lustre-OST0000 complete [ 9320.871982] Lustre: server umount lustre-OST0001 complete [ 9325.237459] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 9331.126545] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9353.245194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9353.379185] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9353.403502] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9353.470743] Lustre: lustre-MDT0000: new disk, initializing [ 9353.525234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9353.528133] Lustre: Skipped 1 previous similar message [ 9353.535843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9356.576671] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9363.630657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9363.712096] Lustre: 241102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9363.756233] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9363.759666] Lustre: Skipped 1 previous similar message [ 9363.828543] Lustre: lustre-MDT0001: new disk, initializing [ 9363.924058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9363.938763] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9366.477099] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9372.371923] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9375.405986] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9378.803764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9378.965096] Lustre: 243069:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9380.044565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9380.048966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9380.094857] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9382.084674] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9388.148972] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9390.626881] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9390.767420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9393.272844] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9393.407123] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9395.695159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9395.695948] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9395.698788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9395.698798] Lustre: Skipped 4 previous similar messages [ 9395.700636] LustreError: Skipped 1 previous similar message [ 9395.725197] Lustre: Skipped 4 previous similar messages [ 9400.854973] Lustre: server umount lustre-OST0000 complete [ 9404.898301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9409.361818] Lustre: server umount lustre-MDT0000 complete [ 9411.045101] LustreError: 241107:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9411.055557] LustreError: 241107:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9411.624135] LustreError: 243068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836332 with bad export cookie 3891808705236689334 [ 9411.627322] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9411.631166] LustreError: 243068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9411.815873] Lustre: server umount lustre-MDT0001 complete [ 9416.680695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9416.994835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9416.998660] Lustre: Skipped 2 previous similar messages [ 9419.149936] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9422.305407] LustreError: 245280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9422.313537] LustreError: 245280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9423.720923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9426.051295] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9430.086766] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9433.205581] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9436.733870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9439.846638] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9443.822223] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9590.756665] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9590.759585] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9590.764486] Lustre: Skipped 2 previous similar messages [ 9590.766401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9590.768520] Lustre: Skipped 4 previous similar messages [ 9593.120897] Lustre: server umount lustre-OST0000 complete [ 9609.184126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9609.249603] Lustre: server umount lustre-MDT0000 complete [ 9610.614290] LustreError: 245260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836531 with bad export cookie 3891808705236690216 [ 9610.616907] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9610.618856] LustreError: 245260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9610.750499] Lustre: server umount lustre-MDT0001 complete [ 9708.841134] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 9710.280372] Key type lgssc unregistered [ 9710.457302] LNet: 248885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9710.465460] LNetError: 248885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9711.527345] LNet: Removed LNI 192.168.201.118@tcp [ 9711.999140] Key type .llcrypt unregistered [ 9712.001205] Key type ._llcrypt unregistered [ 9725.323804] Key type ._llcrypt registered [ 9725.325736] Key type .llcrypt registered [ 9725.384859] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 9733.686493] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9734.151515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9734.210667] alg: No test for adler32 (adler32-zlib) [ 9735.137289] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 9735.279649] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 9736.872334] Key type lgssc registered [ 9737.419753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9759.539787] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9764.469676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9764.481359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9765.599079] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9765.617833] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9765.658593] Lustre: lustre-MDT0000: new disk, initializing [ 9765.697438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9765.708798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9767.732837] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9773.439421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9773.497990] Lustre: 252930:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9773.532649] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9773.535500] Lustre: Skipped 1 previous similar message [ 9773.583338] Lustre: lustre-MDT0001: new disk, initializing [ 9773.649420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9773.672394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9773.677618] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9775.796514] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9778.908805] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9782.295181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9782.441581] Lustre: lustre-OST0000: new disk, initializing [ 9782.443369] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9782.450259] Lustre: 254561:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9782.510730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9785.919981] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9786.888565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9786.891942] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9786.941171] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9791.596072] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9791.684157] Lustre: lustre-OST0001: new disk, initializing [ 9791.686564] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9791.689900] Lustre: 255431:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9791.746070] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9794.588152] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9796.625461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9796.629558] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9796.669251] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9800.737968] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9804.971128] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9809.377119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9809.380284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9809.385318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9811.942351] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9811.942400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9811.948740] Lustre: Skipped 1 previous similar message [ 9811.959299] Lustre: Skipped 3 previous similar messages [ 9815.270187] Lustre: server umount lustre-MDT0000 complete [ 9817.086600] LustreError: 252922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836738 with bad export cookie 11929276514957991144 [ 9817.090826] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9817.093809] LustreError: 252922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9817.288664] Lustre: server umount lustre-MDT0001 complete [ 9829.417223] Lustre: server umount lustre-OST0000 complete [ 9841.657092] Lustre: server umount lustre-OST0001 complete [ 9848.043537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9852.460712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9856.649884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9861.086217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9869.060678] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9873.793333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9873.811683] Lustre: lustre-MDT0000: reset Object Index mappings [ 9874.026904] LustreError: 260432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9874.042132] LustreError: 260432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9874.064534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9875.867705] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9879.521523] LustreError: 260433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9879.864953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9879.873095] Lustre: lustre-MDT0001: reset Object Index mappings [ 9881.919221] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9883.232752] Lustre: 261573:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9885.982611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9886.149737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9886.155393] Lustre: Skipped 1 previous similar message [ 9888.632898] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9890.283162] LustreError: 261926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9892.585085] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9895.597794] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9900.013384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9900.015443] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9904.704905] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9906.407860] Lustre: 263446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9908.196473] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9908.197792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9908.202023] Lustre: Skipped 3 previous similar messages [ 9908.204223] Lustre: Skipped 1 previous similar message [ 9913.315503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9913.319142] Lustre: Skipped 2 previous similar messages [ 9914.123665] Lustre: server umount lustre-MDT0000 complete [ 9915.998616] LustreError: 260412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836837 with bad export cookie 11929276514957997934 [ 9916.003825] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9916.010439] LustreError: 260412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9916.161417] Lustre: server umount lustre-MDT0001 complete [ 9928.652703] Lustre: server umount lustre-OST0000 complete [ 9940.751117] Lustre: server umount lustre-OST0001 complete [ 9949.959545] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 19:34:31 (1786836871) [ 9957.935996] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 9962.544667] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9980.041672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9980.146715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9980.183188] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9980.221252] Lustre: lustre-MDT0000: new disk, initializing [ 9980.260849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9980.264064] Lustre: Skipped 1 previous similar message [ 9980.273460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9982.122729] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9987.266106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9987.318951] Lustre: 267693:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9987.472468] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9989.128769] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9993.299312] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9995.294754] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9997.814828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9997.920045] Lustre: lustre-OST0000: new disk, initializing [ 9997.921488] Lustre: Skipped 1 previous similar message [ 9997.923339] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9997.925503] Lustre: Skipped 2 previous similar messages [ 9997.928602] Lustre: 269659:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9999.942295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9999.945821] Lustre: Skipped 1 previous similar message [ 9999.947937] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9999.991212] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10000.399735] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10004.860566] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10006.997828] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10007.091983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10009.002635] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10009.086620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10010.081544] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10010.084809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10010.090215] Lustre: Skipped 1 previous similar message [10010.092222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10015.975746] Lustre: server umount lustre-OST0000 complete [10018.273404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10018.277578] LustreError: Skipped 1 previous similar message [10018.280100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10018.288942] Lustre: Skipped 1 previous similar message [10018.292846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10018.295569] Lustre: Skipped 3 previous similar messages [10032.096150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10032.172635] Lustre: server umount lustre-MDT0000 complete [10033.935235] LustreError: 268487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836954 with bad export cookie 11929276514958000006 [10033.937627] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10033.943410] LustreError: 268487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10034.106623] Lustre: server umount lustre-MDT0001 complete [10042.666445] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10047.171085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10047.378953] LustreError: 272587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10047.415692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10047.419450] Lustre: Skipped 2 previous similar messages [10049.271798] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10052.584673] LustreError: 272588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10053.491520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10055.586954] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10056.937319] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10060.155807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10062.951753] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10066.631723] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10066.708584] Lustre: lustre-OST0001: new disk, initializing [10066.710852] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10066.714611] Lustre: 274895:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10068.121847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10068.127501] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10068.172579] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10069.400503] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10074.431916] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10075.915711] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10078.002227] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10078.003665] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10082.790051] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10082.791313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10082.794218] Lustre: Skipped 4 previous similar messages [10082.797415] Lustre: Skipped 9 previous similar messages [10086.252574] Lustre: server umount lustre-MDT0000 complete [10087.846385] LustreError: 272570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837008 with bad export cookie 11929276514958000888 [10087.851305] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10087.852320] LustreError: 272570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10087.906556] LustreError: 272582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10087.914243] LustreError: 272582:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10088.022363] Lustre: server umount lustre-MDT0001 complete [10095.813037] Lustre: server umount lustre-OST0000 complete [10097.294473] Lustre: server umount lustre-OST0001 complete [10103.257988] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10107.957203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10108.169374] LustreError: 277946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10109.814512] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10113.078917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10113.237207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10113.239514] Lustre: Skipped 4 previous similar messages [10114.892268] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10116.049226] Lustre: 279086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10118.649066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10121.240905] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10124.816186] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10127.611224] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10133.492082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10133.503445] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10136.758989] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10138.333177] Lustre: 280966:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10140.543846] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10140.555650] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10144.639224] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 19:37:46 (1786837066) [10159.075285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10159.076408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10159.080767] Lustre: Skipped 4 previous similar messages [10159.084944] Lustre: Skipped 5 previous similar messages [10162.669296] Lustre: server umount lustre-MDT0000 complete [10164.197789] LustreError: 277946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10164.206564] LustreError: 277946:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10164.269274] LustreError: 277928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837085 with bad export cookie 11929276514958007188 [10164.273034] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10164.278720] LustreError: 277928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10170.506556] Lustre: server umount lustre-MDT0001 complete [10178.139000] Lustre: server umount lustre-OST0000 complete [10185.837058] Lustre: server umount lustre-OST0001 complete [10188.267548] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [10191.938548] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10211.594209] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10216.018681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10216.135158] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10216.154514] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10216.206237] Lustre: lustre-MDT0000: new disk, initializing [10216.248024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10218.034705] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10223.201790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10223.254881] Lustre: 285198:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10223.376663] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10225.427579] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10231.253450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10231.396831] Lustre: 286829:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10233.048035] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10234.092390] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10239.253495] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10241.146639] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10241.929609] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10247.216497] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10248.640880] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10248.643274] Lustre: Skipped 1 previous similar message [10253.273568] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 19:39:34 (1786837174) [10254.308339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10254.311298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10254.315313] Lustre: Skipped 3 previous similar messages [10254.317391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10254.319798] Lustre: Skipped 2 previous similar messages [10260.215057] Lustre: server umount lustre-MDT0000 complete [10261.602633] LustreError: 285189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837182 with bad export cookie 11929276514958015056 [10261.606383] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10261.606933] LustreError: 285189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10261.742484] Lustre: server umount lustre-MDT0001 complete [10273.567908] Lustre: server umount lustre-OST0000 complete [10285.359435] Lustre: server umount lustre-OST0001 complete [10289.049865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10289.252540] LustreError: 289941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10289.259217] LustreError: 289941:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10289.293332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10289.296749] Lustre: Skipped 6 previous similar messages [10291.016319] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10294.843853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10296.696743] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10299.990630] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10302.430984] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10305.363455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10307.970933] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10311.141148] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10326.489410] loop1: detected capacity change from 0 to 204800000 [10326.560332] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10326.565989] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10326.570221] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10328.314757] loop1: detected capacity change from 0 to 204800000 [10330.318413] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10330.352192] LustreError: 289929:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs5033: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10330.358048] LustreError: 289929:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs5033 [10330.360208] LustreError: 289929:0:(mgs_handler.c:517:mgs_target_reg()) fs5033-MDT0001: Error checking target index: rc = -13 [10330.363856] LustreError: fs5033-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10330.367372] LustreError: 293676:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10330.370662] LustreError: 293676:0:(tgt_mount.c:2129:server_put_super()) no obd fs5033-MDT0001 [10330.372689] LustreError: 293676:0:(tgt_mount.c:116:server_deregister_mount()) fs5033-MDT0001 not registered [10330.384220] Lustre: server umount fs5033-MDT0001 complete [10330.386434] LustreError: 293676:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10333.454994] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10333.483799] Lustre: Setting parameter fs5033-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs5033-MDT0001 [10333.501432] Lustre: srv-fs5033-MDT0001: No data found on store. Initialize space. [10333.503504] Lustre: Skipped 4 previous similar messages [10333.539924] Lustre: fs5033-MDT0001: new disk, initializing [10333.541939] Lustre: Skipped 3 previous similar messages [10333.580071] Lustre: srv-fs5033-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10333.582942] Lustre: 294141:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5033-MDT0001-mdtlov: can't create new llog: rc = -115 [10333.586940] LustreError: 294141:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5033-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10335.003694] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10338.110146] Lustre: server umount fs5033-MDT0001 complete [10343.402743] loop1: detected capacity change from 0 to 204800000 [10345.961702] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10346.108817] Lustre: 295708:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5033-MDT0001-mdtlov: can't create new llog: rc = -115 [10346.113520] LustreError: 295708:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5033-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10348.183933] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10350.523325] Lustre: server umount fs5033-MDT0001 complete [10356.707780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10356.707938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10356.720340] Lustre: Skipped 5 previous similar messages [10360.607896] Lustre: server umount lustre-MDT0000 complete [10363.803478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10363.858793] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10365.682081] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10368.849457] loop1: detected capacity change from 0 to 204800000 [10369.003061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10371.076041] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10371.225377] Lustre: 298147:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5033-MDT0001-mdtlov: can't create new llog: rc = -115 [10371.229379] LustreError: 298147:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5033-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10372.732046] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10374.625497] Lustre: server umount fs5033-MDT0001 complete [10382.304861] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10384.358224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10384.361019] Lustre: Skipped 10 previous similar messages [10388.373758] Lustre: server umount lustre-OST0000 complete [10395.903457] Lustre: server umount lustre-MDT0000 complete [10397.637842] LustreError: 289921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837318 with bad export cookie 11929276514958017730 [10397.640452] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10397.644666] LustreError: 289921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10397.830942] Lustre: server umount lustre-MDT0001 complete [10402.298243] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [10403.525562] Key type lgssc unregistered [10403.669291] LNet: 300491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10403.677215] LNetError: 300491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10403.687402] LNet: Removed LNI 192.168.201.118@tcp [10404.095277] Key type .llcrypt unregistered [10404.096902] Key type ._llcrypt unregistered [10412.746169] Key type ._llcrypt registered [10412.747803] Key type .llcrypt registered [10412.803216] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10413.178131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10413.215056] alg: No test for adler32 (adler32-zlib) [10414.084261] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [10414.203310] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [10415.808213] Key type lgssc registered [10416.264183] Lustre: Echo OBD driver; http://www.lustre.org/ [10419.480189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10419.499401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10420.714342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10422.330164] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10425.732945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10425.923331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10427.585288] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10430.602335] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10432.839819] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10435.691575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10435.850112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10438.157862] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10442.788454] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10445.094048] LustreError: 303566:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10445.098822] LustreError: 303566:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10449.131686] LustreError: 304237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10450.218344] LustreError: 303567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10450.227457] LustreError: 303567:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10454.266498] LustreError: 303565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10459.385580] LustreError: 303565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10459.395387] LustreError: 303565:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10459.661483] loop1: detected capacity change from 0 to 204800000 [10459.735645] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10459.742306] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10459.750303] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10461.868929] loop1: detected capacity change from 0 to 204800000 [10464.313276] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10464.376391] Lustre: fs26261-OST0000: new disk, initializing [10464.378466] Lustre: srv-fs26261-OST0000: No data found on store. Initialize space. [10464.381839] Lustre: 305493:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10464.416984] Lustre: fs26261-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10466.944943] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10469.628809] LustreError: 304240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10469.636863] LustreError: 304240:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10471.499905] Lustre: server umount fs26261-OST0000 complete [10476.933099] loop1: detected capacity change from 0 to 204800000 [10479.667706] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10479.756135] Lustre: fs26261-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10482.378360] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10485.965563] loop2: detected capacity change from 0 to 204800000 [10486.057704] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10486.061544] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10486.066414] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10486.244100] LustreError: 304240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10486.250991] LustreError: 304240:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [10488.290670] loop2: detected capacity change from 0 to 204800000 [10490.934756] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10490.979257] LustreError: fs26261-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10490.985422] LustreError: fs26261-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10490.990339] LustreError: 308530:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10490.994197] LustreError: 308530:0:(tgt_mount.c:2129:server_put_super()) no obd fs26261-OST0001 [10490.996683] LustreError: 308530:0:(tgt_mount.c:116:server_deregister_mount()) fs26261-OST0001 not registered [10491.019297] Lustre: server umount fs26261-OST0001 complete [10491.021517] LustreError: 308530:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10495.543183] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10495.598349] Lustre: fs26261-OST0001: new disk, initializing [10495.600506] Lustre: srv-fs26261-OST0001: No data found on store. Initialize space. [10495.603608] Lustre: 309082:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10495.632852] Lustre: fs26261-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10497.946195] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10501.918305] Lustre: server umount fs26261-OST0000 complete [10505.384751] Lustre: server umount fs26261-OST0001 complete [10509.599214] loop1: detected capacity change from 0 to 204800000 [10511.508850] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10513.594284] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10515.972518] loop2: detected capacity change from 0 to 204800000 [10517.899300] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10517.965275] Lustre: fs26261-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10517.968451] Lustre: Skipped 1 previous similar message [10519.973274] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10520.825611] LustreError: 303565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10520.831209] LustreError: 303565:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [10523.009607] Lustre: server umount fs26261-OST0000 complete [10526.435082] Lustre: server umount fs26261-OST0001 complete [10532.325527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10532.326725] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10532.328403] LustreError: Skipped 1 previous similar message [10532.332907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10532.336104] Lustre: Skipped 1 previous similar message [10536.736190] Lustre: server umount lustre-OST0000 complete [10538.465154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10538.468675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10538.474442] Lustre: Skipped 1 previous similar message [10538.478137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10542.561839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10542.562274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10544.482194] Lustre: server umount lustre-MDT0000 complete [10545.911189] LustreError: 309081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837466 with bad export cookie 6263392230039292647 [10545.914938] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10545.917382] LustreError: 309081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10546.048233] Lustre: server umount lustre-MDT0001 complete [10549.977435] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [10551.029728] Key type lgssc unregistered [10551.170372] LNet: 314995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10551.173536] LNetError: 314995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10551.180478] LNet: Removed LNI 192.168.201.118@tcp [10551.489125] Key type .llcrypt unregistered [10551.490620] Key type ._llcrypt unregistered [10562.868435] Key type ._llcrypt registered [10562.869668] Key type .llcrypt registered [10562.906110] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [10570.061944] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10570.574585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10570.597115] alg: No test for adler32 (adler32-zlib) [10571.466548] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [10571.595042] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [10573.208210] Key type lgssc registered [10573.796026] Lustre: Echo OBD driver; http://www.lustre.org/ [10587.838301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10587.847920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10588.949052] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10588.965318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10589.009815] Lustre: lustre-MDT0000: new disk, initializing [10589.042782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10589.051496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10590.588078] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10595.251708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10595.304209] Lustre: 318824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10595.333392] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10595.336071] Lustre: Skipped 1 previous similar message [10595.372351] Lustre: lustre-MDT0001: new disk, initializing [10595.411746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10595.426543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10595.431565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10596.912093] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10601.523102] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10603.688433] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10606.324803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10606.438925] Lustre: lustre-OST0000: new disk, initializing [10606.441232] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10606.444283] Lustre: 320794:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10606.477873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10608.582940] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10612.684294] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10613.251035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10613.254879] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10613.305661] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10613.538383] Lustre: Mounted lustre-client - version 2.17.57_1_g2c0d074 [10613.936921] Lustre: Unmounted lustre-client [10618.337770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10618.337957] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10618.341061] LustreError: Skipped 1 previous similar message [10618.346398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10618.349934] Lustre: Skipped 1 previous similar message [10619.287777] Lustre: server umount lustre-OST0000 complete [10620.896796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10620.899404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10620.903782] Lustre: Skipped 1 previous similar message [10620.906756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10623.969705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10623.970111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10623.977356] Lustre: Skipped 1 previous similar message [10626.716549] Lustre: server umount lustre-MDT0000 complete [10628.048364] LustreError: 320793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837549 with bad export cookie 13448668452341392795 [10628.051622] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10628.052881] LustreError: 320793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10628.169921] Lustre: server umount lustre-MDT0001 complete [10632.213457] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [10633.298782] Key type lgssc unregistered [10663.776885] LNet: 1 local NIs in recovery (showing 1): 192.168.201.118@tcp [10664.161846] LNet: 322954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10664.167285] LNetError: 322954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10745.062278] LNet: Removed LNI 192.168.201.118@tcp [10745.382475] Key type .llcrypt unregistered [10745.383511] Key type ._llcrypt unregistered [10751.252916] Key type ._llcrypt registered [10751.253973] Key type .llcrypt registered [10751.290415] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [10757.434356] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10757.835076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10757.841414] alg: No test for adler32 (adler32-zlib) [10758.699151] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [10758.801168] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [10760.384133] Key type lgssc registered [10760.790536] Lustre: Echo OBD driver; http://www.lustre.org/ [10770.893047] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 19:48:12 (1786837692) [10773.189897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10773.196542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10774.278425] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10774.291579] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10774.329547] Lustre: lustre-MDT0000: new disk, initializing [10774.355357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10774.362208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10775.798951] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10780.045910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10780.092074] Lustre: 326154:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10780.114143] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10780.116392] Lustre: Skipped 1 previous similar message [10780.151923] Lustre: lustre-MDT0001: new disk, initializing [10780.184492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10780.196815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10780.200677] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10781.607501] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10785.588089] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10787.884106] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10790.764458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10790.872652] Lustre: lustre-OST0000: new disk, initializing [10790.874665] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10790.878232] Lustre: 328121:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10790.916188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10792.997483] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10796.980743] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10800.123069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10800.126458] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10800.159765] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10810.338823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10810.339532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10810.343434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10810.347669] Lustre: Skipped 1 previous similar message [10813.844529] Lustre: server umount lustre-OST0000 complete [10815.457428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10815.458767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10815.461669] Lustre: Skipped 2 previous similar messages [10815.466122] Lustre: Skipped 1 previous similar message [10820.577541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10820.579656] Lustre: Skipped 1 previous similar message [10821.341647] Lustre: server umount lustre-MDT0000 complete [10822.660757] LustreError: 326147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837743 with bad export cookie 2578197600175352560 [10822.663220] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10822.664763] LustreError: 326147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10822.798094] Lustre: server umount lustre-MDT0001 complete [10826.576655] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [10827.655931] Key type lgssc unregistered [10827.797462] LNet: 330029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10827.800374] LNetError: 330029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10827.810361] LNet: Removed LNI 192.168.201.118@tcp [10828.137500] Key type .llcrypt unregistered [10828.138905] Key type ._llcrypt unregistered [10836.886940] Key type ._llcrypt registered [10836.888096] Key type .llcrypt registered [10836.937191] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10837.342392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10837.384892] alg: No test for adler32 (adler32-zlib) [10838.280507] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [10838.387820] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [10839.976132] Key type lgssc registered [10840.397868] Lustre: Echo OBD driver; http://www.lustre.org/ [10843.268832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10843.275827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10844.472762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10846.076439] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10849.491437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10849.655111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10851.432700] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10854.654358] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10856.749494] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10859.259745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10859.388239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10861.536276] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10864.361195] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10873.271917] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10873.874073] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10875.360991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10875.365801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10875.371536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10879.969247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10879.970084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10879.972545] Lustre: Skipped 1 previous similar message [10879.978857] Lustre: Skipped 1 previous similar message [10881.500346] Lustre: server umount lustre-MDT0000 complete [10882.919501] LustreError: 331262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837803 with bad export cookie 12717201210826498715 [10882.922560] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10882.924060] LustreError: 331262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10885.088902] LustreError: 331274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10885.089214] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10885.093504] LustreError: 331274:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10885.098068] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10885.100880] Lustre: Skipped 2 previous similar messages [10887.139753] LustreError: 331304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10887.140045] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10887.148471] Lustre: Skipped 1 previous similar message [10887.149251] LustreError: 331304:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10890.488316] LustreError: 331274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10892.257959] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10892.261022] Lustre: Skipped 1 previous similar message [10895.609412] LustreError: 331275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10895.614758] LustreError: 331275:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10897.376152] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10897.466208] Lustre: server umount lustre-MDT0001 complete [10904.851452] Lustre: server umount lustre-OST0000 complete [10907.723626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10909.123756] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10910.102163] Lustre: 335458:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.118@tcp [10910.818698] Lustre: server umount MGS complete [10913.767537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10913.917506] LustreError: 336028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10913.926755] LustreError: 336028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10913.951379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10915.345724] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10918.334947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10918.463975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10919.868629] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10922.919743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10925.089284] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10927.653590] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10928.177229] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [10930.072724] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10930.644414] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10933.221168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10933.222146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10933.225262] Lustre: Skipped 1 previous similar message [10933.227193] Lustre: Skipped 2 previous similar messages [10933.230671] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10934.742901] Lustre: server umount lustre-OST0000 complete [10938.265163] Lustre: server umount lustre-MDT0000 complete [10938.338357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10938.345177] LustreError: 337482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10938.351460] LustreError: 337482:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10939.627181] LustreError: 336008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837860 with bad export cookie 12717201210826499541 [10939.630884] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10939.631018] LustreError: 336008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10939.746164] Lustre: server umount lustre-MDT0001 complete [10943.756036] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [10944.834359] Key type lgssc unregistered [10944.967485] LNet: 339461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10944.970751] LNetError: 339461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10944.983377] LNet: Removed LNI 192.168.201.118@tcp [10945.323291] Key type .llcrypt unregistered [10945.324474] Key type ._llcrypt unregistered [10953.654782] Key type ._llcrypt registered [10953.655694] Key type .llcrypt registered [10953.691229] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [10960.196777] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10960.519584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10960.549080] alg: No test for adler32 (adler32-zlib) [10961.405748] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [10961.491183] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [10963.072182] Key type lgssc registered [10963.460373] Lustre: Echo OBD driver; http://www.lustre.org/ [10975.885735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10975.891373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10976.970879] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10976.981241] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10977.017708] Lustre: lustre-MDT0000: new disk, initializing [10977.040402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10977.048071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10978.413505] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10982.702231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10982.742768] Lustre: 342594:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10982.773963] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10982.774683] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10982.779302] Lustre: Skipped 1 previous similar message [10982.818798] Lustre: lustre-MDT0001: new disk, initializing [10982.852564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10982.872717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10982.877303] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10984.397640] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10988.323975] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10990.223352] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10992.452467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10992.548679] Lustre: lustre-OST0000: new disk, initializing [10992.550490] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10992.553491] Lustre: 344563:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10992.584266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10994.443020] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10997.243782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10997.246974] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10997.282069] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10998.227689] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10999.646198] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10999.732658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11001.223182] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11001.306235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11002.338404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11002.339632] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11002.340183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11002.340606] LustreError: Skipped 1 previous similar message [11002.343564] Lustre: Skipped 1 previous similar message [11004.054504] Lustre: server umount lustre-OST0000 complete [11007.457477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11007.458180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11007.460484] Lustre: Skipped 1 previous similar message [11007.463716] Lustre: Skipped 2 previous similar messages [11011.479840] Lustre: server umount lustre-MDT0000 complete [11012.665141] LustreError: 342587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837933 with bad export cookie 17430775696696299614 [11012.667831] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11012.668664] LustreError: 342587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11012.780311] Lustre: server umount lustre-MDT0001 complete [11015.337579] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 19:52:17 (1786837937) [11020.179705] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [11023.462305] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [11035.277042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11035.354553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11035.364082] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11035.400383] Lustre: lustre-MDT0000: new disk, initializing [11035.421670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11035.428219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11036.795063] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11040.996299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11041.046960] Lustre: 349310:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11041.072952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11041.075456] Lustre: Skipped 1 previous similar message [11041.106246] Lustre: lustre-MDT0001: new disk, initializing [11041.138642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11041.150276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11041.153792] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11042.462626] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11046.206511] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11048.147476] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11050.704467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11050.801896] Lustre: lustre-OST0000: new disk, initializing [11050.804350] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11050.807602] Lustre: 351278:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11050.839339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11053.083085] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11057.019281] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11058.682669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11058.686729] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11058.722109] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11074.021068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11074.021321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11074.023236] LustreError: Skipped 1 previous similar message [11074.023729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11074.028761] Lustre: Skipped 1 previous similar message [11074.034182] Lustre: Skipped 1 previous similar message [11077.924972] Lustre: server umount lustre-OST0000 complete [11082.209797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11082.213123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11082.220152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11085.488701] Lustre: server umount lustre-MDT0000 complete [11086.894161] LustreError: 350104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786838007 with bad export cookie 17430775696696300496 [11086.898043] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11086.898400] LustreError: 350104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11087.037485] Lustre: server umount lustre-MDT0001 complete [11091.130838] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [11092.213698] Key type lgssc unregistered [11092.366477] LNet: 353430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11092.370326] LNetError: 353430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11092.380530] LNet: Removed LNI 192.168.201.118@tcp [11092.679809] Key type .llcrypt unregistered [11092.681406] Key type ._llcrypt unregistered [11126.836891] Key type ._llcrypt registered [11126.838129] Key type .llcrypt registered [11126.875917] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [11127.005324] Key type .llcrypt unregistered [11127.006744] Key type ._llcrypt unregistered [11154.110220] Key type ._llcrypt registered [11154.111211] Key type .llcrypt registered [11154.146635] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [11154.246182] Key type .llcrypt unregistered [11154.247313] Key type ._llcrypt unregistered [11165.197313] Key type ._llcrypt registered [11165.198551] Key type .llcrypt registered [11165.241777] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [11165.370302] Key type .llcrypt unregistered [11165.371562] Key type ._llcrypt unregistered [11175.927814] Key type ._llcrypt registered [11175.929272] Key type .llcrypt registered [11175.962640] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [11176.071285] Key type .llcrypt unregistered [11176.072941] Key type ._llcrypt unregistered [11202.257536] Key type ._llcrypt registered [11202.258468] Key type .llcrypt registered [11202.294050] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [11202.401448] Key type .llcrypt unregistered [11202.402659] Key type ._llcrypt unregistered [11213.148187] Key type ._llcrypt registered [11213.148990] Key type .llcrypt registered [11213.183624] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [11213.292656] Key type .llcrypt unregistered [11213.294203] Key type ._llcrypt unregistered [11223.879589] Key type ._llcrypt registered [11223.880521] Key type .llcrypt registered [11223.914760] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [11224.017244] Key type .llcrypt unregistered [11224.018682] Key type ._llcrypt unregistered [11234.503383] Key type ._llcrypt registered [11234.504921] Key type .llcrypt registered [11234.541519] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [11234.660515] Key type .llcrypt unregistered [11234.661501] Key type ._llcrypt unregistered [11245.537889] Key type ._llcrypt registered [11245.539462] Key type .llcrypt registered [11245.584914] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [11245.696712] Key type .llcrypt unregistered [11245.698400] Key type ._llcrypt unregistered [11256.555229] Key type ._llcrypt registered [11256.556195] Key type .llcrypt registered [11256.590173] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [11256.700387] Key type .llcrypt unregistered [11256.701938] Key type ._llcrypt unregistered [11277.849214] Key type ._llcrypt registered [11277.850570] Key type .llcrypt registered [11277.889829] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [11278.004379] Key type .llcrypt unregistered [11278.005785] Key type ._llcrypt unregistered [11289.556965] Key type ._llcrypt registered [11289.558135] Key type .llcrypt registered [11289.599486] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [11289.719419] Key type .llcrypt unregistered [11289.720628] Key type ._llcrypt unregistered [11300.892313] Key type ._llcrypt registered [11300.893641] Key type .llcrypt registered [11300.930289] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [11301.039248] Key type .llcrypt unregistered [11301.040789] Key type ._llcrypt unregistered [11319.330387] Key type ._llcrypt registered [11319.331365] Key type .llcrypt registered [11319.365025] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [11325.539642] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [11325.839128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11325.845200] alg: No test for adler32 (adler32-zlib) [11326.693694] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [11326.777753] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [11328.352148] Key type lgssc registered [11328.691214] Lustre: Echo OBD driver; http://www.lustre.org/ [11339.766055] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11339.772425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11340.849919] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [11340.861085] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [11340.897046] Lustre: tst250fs-MDT0000: new disk, initializing [11340.920107] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11340.927221] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11342.290848] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11346.276736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11346.316196] Lustre: 373488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11346.337858] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [11346.339484] Lustre: Skipped 1 previous similar message [11346.377030] Lustre: tst250fs-MDT0001: new disk, initializing [11346.408846] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11346.424036] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11346.427979] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11347.749832] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11350.007364] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11352.118922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11352.217053] Lustre: tst250fs-OST0000: new disk, initializing [11352.218941] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [11352.222203] Lustre: 375116:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11352.257514] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11354.078449] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11355.513045] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11355.515885] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11355.552304] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11358.213995] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11358.270235] Lustre: tst250fs-OST0001: new disk, initializing [11358.272583] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [11358.274845] Lustre: 375991:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11358.306672] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11360.149103] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11361.538398] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11361.541639] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11361.572235] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11362.626025] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [11375.012171] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11375.015455] Lustre: Skipped 2 previous similar messages [11375.016318] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11375.018311] Lustre: Skipped 2 previous similar messages [11378.655428] Lustre: server umount tst250fs-MDT0000 complete [11379.915246] LustreError: 374281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786838300 with bad export cookie 11604927825854633161 [11379.919080] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11379.920104] LustreError: 374281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11380.039218] Lustre: server umount tst250fs-MDT0001 complete [11391.621951] Lustre: server umount tst250fs-OST0000 complete [11402.152241] Lustre: server umount tst250fs-OST0001 complete [11408.298051] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [11411.029748] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [11420.675663] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 19:59:02 (1786838342) [11421.189791] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [11426.316898] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [11429.689929] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [11439.698284] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11098 sec ========= 19:59:21 (1786838361) [11440.292926] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:59:21 (1786838361) === [11441.466792] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:59:23 (1786838363) === [11454.033293] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [11454.950335] Key type lgssc unregistered [11455.066297] LNet: 385102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11455.068408] LNetError: 385102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11455.077283] LNet: Removed LNI 192.168.201.118@tcp [11455.378130] Key type .llcrypt unregistered [11455.379839] Key type ._llcrypt unregistered