[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 489468521 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002326] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008008] pid_max: default: 32768 minimum: 301 [ 0.009134] LSM: Security Framework initializing [ 0.010050] Yama: becoming mindful. [ 0.011030] SELinux: Initializing. [ 0.012049] *** VALIDATE selinux *** [ 0.019763] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023416] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025168] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027021] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029134] *** VALIDATE tmpfs *** [ 0.030518] *** VALIDATE proc *** [ 0.031264] *** VALIDATE cgroup *** [ 0.032007] *** VALIDATE cgroup2 *** [ 0.033259] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036036] Spectre V2 : User space: Vulnerable [ 0.037007] Speculative Store Bypass: Vulnerable [ 0.040371] debug: unmapping init [mem 0xffffffffb1259000-0xffffffffb1260fff] [ 0.043000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043721] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044030] ... version: 2 [ 0.045013] ... bit width: 48 [ 0.046010] ... generic registers: 4 [ 0.047011] ... value mask: 0000ffffffffffff [ 0.048013] ... max period: 00007fffffffffff [ 0.049011] ... fixed-purpose events: 3 [ 0.050011] ... event mask: 000000070000000f [ 0.052311] rcu: Hierarchical SRCU implementation. [ 0.054786] smp: Bringing up secondary CPUs ... [ 0.055632] x86: Booting SMP configuration: [ 0.056032] .... node #0, CPUs: #1 #2 #3 [ 0.064039] smp: Brought up 1 node, 4 CPUs [ 0.066021] smpboot: Max logical packages: 1 [ 0.067076] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.120840] node 0 deferred pages initialised in 52ms [ 0.124125] devtmpfs: initialized [ 0.126347] x86/mm: Memory block size: 128MB [ 0.130630] gcov: version magic: 0x41383552 [ 0.133189] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.136096] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.140563] pinctrl core: initialized pinctrl subsystem [ 0.142188] [ 0.142858] ************************************************************* [ 0.145017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150019] ** ** [ 0.157020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160013] ** ** [ 0.161008] ** This means that this kernel is built to expose internal ** [ 0.164012] ** IOMMU data structures, which may compromise security on ** [ 0.166042] ** your system. ** [ 0.168012] ** ** [ 0.170012] ** If you see this message and you are not debugging the ** [ 0.172010] ** kernel, report this immediately to your vendor! ** [ 0.174011] ** ** [ 0.176011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.178011] ************************************************************* [ 0.181788] NET: Registered protocol family 16 [ 0.183367] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.185044] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.187071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.191113] cpuidle: using governor menu [ 0.193062] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.194852] PCI: Using configuration type 1 for base access [ 0.196141] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.203122] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.205017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.208063] cryptd: max_cpu_qlen set to 1000 [ 0.209283] ACPI: Added _OSI(Module Device) [ 0.211015] ACPI: Added _OSI(Processor Device) [ 0.213011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.214011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.221371] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.232213] ACPI: Interpreter enabled [ 0.233068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.236022] ACPI: Using IOAPIC for interrupt routing [ 0.239195] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.243487] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.252000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.254042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.256014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.259100] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.260000] acpiphp: Slot [2] registered [ 0.260000] acpiphp: Slot [5] registered [ 0.260130] acpiphp: Slot [6] registered [ 0.261000] acpiphp: Slot [7] registered [ 0.263084] acpiphp: Slot [8] registered [ 0.265131] acpiphp: Slot [9] registered [ 0.267217] acpiphp: Slot [10] registered [ 0.269110] acpiphp: Slot [3] registered [ 0.271085] acpiphp: Slot [4] registered [ 0.272085] acpiphp: Slot [11] registered [ 0.274165] acpiphp: Slot [12] registered [ 0.275092] acpiphp: Slot [13] registered [ 0.278106] acpiphp: Slot [14] registered [ 0.280126] acpiphp: Slot [15] registered [ 0.282018] acpiphp: Slot [16] registered [ 0.283108] acpiphp: Slot [17] registered [ 0.287143] acpiphp: Slot [18] registered [ 0.289112] acpiphp: Slot [19] registered [ 0.292217] acpiphp: Slot [20] registered [ 0.294218] acpiphp: Slot [21] registered [ 0.296078] acpiphp: Slot [22] registered [ 0.298080] acpiphp: Slot [23] registered [ 0.299109] acpiphp: Slot [24] registered [ 0.301092] acpiphp: Slot [25] registered [ 0.303069] acpiphp: Slot [26] registered [ 0.304085] acpiphp: Slot [27] registered [ 0.306100] acpiphp: Slot [28] registered [ 0.307068] acpiphp: Slot [29] registered [ 0.309077] acpiphp: Slot [30] registered [ 0.310091] acpiphp: Slot [31] registered [ 0.312084] PCI host bridge to bus 0000:00 [ 0.314016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.317031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.320024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.323032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.326030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.330040] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.333313] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.336405] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.343252] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.355016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.360048] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.362012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.364012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.365011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.369120] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.371894] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.375035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.379022] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.383011] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.393015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.396999] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.400643] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.406011] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.410000] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.427013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.438000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.444015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.449015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.470013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.480310] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.486011] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.496011] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.526024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.540743] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.548016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.555035] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.578012] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.594000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.609019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.619026] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.648039] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.670050] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.678017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.688018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.707023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.730289] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.735009] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.740546] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.745790] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.747262] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.753111] iommu: Default domain type: Passthrough [ 0.755480] SCSI subsystem initialized [ 0.757105] ACPI: bus type USB registered [ 0.758073] usbcore: registered new interface driver usbfs [ 0.760083] usbcore: registered new interface driver hub [ 0.761048] usbcore: registered new device driver usb [ 0.762081] pps_core: LinuxPPS API ver. 1 registered [ 0.763006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.765030] PTP clock support registered [ 0.767387] EDAC MC: Ver: 3.0.0 [ 0.768322] PCI: Using ACPI for IRQ routing [ 0.769457] NetLabel: Initializing [ 0.770005] NetLabel: domain hash size = 128 [ 0.770917] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.773067] NetLabel: unlabeled traffic allowed by default [ 0.775002] vgaarb: loaded [ 0.776246] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.777004] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.783512] clocksource: Switched to clocksource kvm-clock [ 0.902734] VFS: Disk quotas dquot_6.6.0 [ 0.903975] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.905917] *** VALIDATE ramfs *** [ 0.906872] *** VALIDATE hugetlbfs *** [ 0.908036] pnp: PnP ACPI init [ 0.909940] pnp: PnP ACPI: found 6 devices [ 0.925402] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.927893] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.929583] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.932018] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.933920] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.935620] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.938805] NET: Registered protocol family 2 [ 0.940993] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.945625] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.949852] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.956766] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.960707] TCP: Hash tables configured (established 65536 bind 65536) [ 0.964290] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.967845] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.971674] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.974882] NET: Registered protocol family 1 [ 0.978557] RPC: Registered named UNIX socket transport module. [ 0.980786] RPC: Registered udp transport module. [ 0.982175] RPC: Registered tcp transport module. [ 0.983692] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.985886] NET: Registered protocol family 44 [ 0.987756] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.990310] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.992561] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.995181] PCI: CLS 0 bytes, default 64 [ 0.997440] Unpacking initramfs... [ 2.687695] debug: unmapping init [mem 0xffff8d537cc54000-0xffff8d537ffbffff] [ 2.691777] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.693686] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.696329] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.207915] Initialise system trusted keyrings [ 3.209451] Key type blacklist registered [ 3.211488] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.220559] zbud: loaded [ 3.223197] *** VALIDATE nfs *** [ 3.224336] *** VALIDATE nfs4 *** [ 3.225954] pstore: using deflate compression [ 3.229203] Platform Keyring initialized [ 3.334328] NET: Registered protocol family 38 [ 3.336103] Key type asymmetric registered [ 3.337441] Asymmetric key parser 'x509' registered [ 3.338982] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.341437] io scheduler mq-deadline registered [ 3.342683] io scheduler kyber registered [ 3.343867] io scheduler bfq registered [ 3.345485] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.348190] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.350712] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.353350] ACPI: Power Button [PWRF] [ 3.358480] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.366233] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.385765] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.396910] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.413844] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.450359] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.490523] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.499903] Non-volatile memory driver v1.3 [ 3.501235] Linux agpgart interface v0.103 [ 3.534038] virtio_blk virtio1: [vda] 146656 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.536347] vda: detected capacity change from 0 to 75087872 [ 3.551246] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.555441] vdb: detected capacity change from 0 to 1073741824 [ 3.569826] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.574582] vdc: detected capacity change from 0 to 2621440000 [ 3.593583] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.598941] vdd: detected capacity change from 0 to 2621440000 [ 3.619475] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.621886] vde: detected capacity change from 0 to 4294967296 [ 3.641412] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.645603] vdf: detected capacity change from 0 to 4294967296 [ 3.653653] libphy: Fixed MDIO Bus: probed [ 3.666898] usbcore: registered new interface driver usbserial_generic [ 3.669407] usbserial: USB Serial support registered for generic [ 3.671982] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.678329] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.680698] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.683185] mousedev: PS/2 mouse device common for all mice [ 3.686512] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.692070] rtc_cmos 00:05: RTC can wake from S4 [ 3.699089] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.701285] rtc_cmos 00:05: registered as rtc0 [ 3.712983] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.715868] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.715939] intel_pstate: CPU model not supported [ 3.724925] hid: raw HID events driver (C) Jiri Kosina [ 3.730390] usbcore: registered new interface driver usbhid [ 3.732131] usbhid: USB HID core driver [ 3.736402] drop_monitor: Initializing network drop monitor service [ 3.738615] Initializing XFRM netlink socket [ 3.740859] NET: Registered protocol family 10 [ 3.744412] Segment Routing with IPv6 [ 3.745645] NET: Registered protocol family 17 [ 3.747336] mpls_gso: MPLS GSO support [ 3.755250] RAS: Correctable Errors collector initialized. [ 3.757356] AVX version of gcm_enc/dec engaged. [ 3.758849] AES CTR mode by8 optimization enabled [ 3.846420] sched_clock: Marking stable (3846403145, 0)->(4729724214, -883321069) [ 3.849876] registered taskstats version 1 [ 3.855284] Loading compiled-in X.509 certificates [ 3.857752] zswap: loaded using pool lzo/zbud [ 3.890774] Key type big_key registered [ 3.939044] Key type encrypted registered [ 3.942298] ima: No TPM chip found, activating TPM-bypass! [ 3.949054] ima: Allocated hash algorithm: sha1 [ 3.953313] ima: No architecture policies found [ 3.954969] evm: Initialising EVM extended attributes: [ 3.958341] evm: security.selinux [ 3.960869] evm: security.ima [ 3.963279] evm: security.capability [ 3.965762] evm: HMAC attrs: 0x1 [ 3.972667] rtc_cmos 00:05: setting system clock to 2026-09-13 01:39:22 UTC (1789263562) [ 3.987643] debug: unmapping init [mem 0xffffffffb2203000-0xffffffffb23fffff] [ 3.992246] debug: unmapping init [mem 0xffffffffb0f82000-0xffffffffb1258fff] [ 4.001912] Write protecting the kernel read-only data: 28672k [ 4.010688] debug: unmapping init [mem 0xffffffffaf603000-0xffffffffaf7fffff] [ 4.015370] debug: unmapping init [mem 0xffffffffaff14000-0xffffffffafffffff] [ 4.051728] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.060965] systemd[1]: Detected virtualization kvm. [ 4.062696] systemd[1]: Detected architecture x86-64. [ 4.064540] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.091832] systemd[1]: No hostname configured. [ 4.093431] systemd[1]: Set hostname to . [ 4.095270] random: systemd: uninitialized urandom read (16 bytes read) [ 4.097310] systemd[1]: Initializing machine ID from random generator. [ 4.188967] random: ln: uninitialized urandom read (6 bytes read) [ 4.313334] random: systemd: uninitialized urandom read (16 bytes read) [ 4.315598] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.320342] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.325418] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.470337] device-mapper: uevent: version 1.0.3 [ 5.474300] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.947187] virtio_net virtio0 ens2: renamed from eth0 [ 6.993513] random: fast init done [ 7.116274] scsi host0: ata_piix [ 7.177021] scsi host1: ata_piix [ 7.184586] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.188437] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.069572] dracut-initqueue[585]: RTNETLINK answers: File exists [ 12.476826] random: crng init done [ 12.478066] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 13.134750] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.880387] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.355879] SELinux: Disabled at runtime. [ 15.455856] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.464199] systemd[1]: Detected virtualization kvm. [ 15.466777] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.352377] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.356873] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.365279] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.375311] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.380526] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.391759] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.402142] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ 16.459087] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 17.139107] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.505188] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.628995] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 17.842261] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 17.885670] EDAC sbridge: Ver: 1.1.2 [ 21.207985] Key type dns_resolver registered [ 21.877666] NFS: Registering the id_resolver key type [ 21.881799] Key type id_resolver registered [ 21.883589] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg410-server login: [ 45.968265] libcfs: loading out-of-tree module taints kernel. [ 45.984062] Key type ._llcrypt registered [ 45.985133] Key type .llcrypt registered [ 46.020831] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid [ 54.155520] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 54.811014] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 54.818947] alg: No test for adler32 (adler32-zlib) [ 55.883347] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 56.228821] LNet: Added LNI 192.168.204.110@tcp [8/256/0/180] [ 57.863376] Key type lgssc registered [ 58.509531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 66.239533] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 81.150989] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 85.659763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 85.670450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 86.754363] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 86.764739] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 86.801830] Lustre: lustre-MDT0000: new disk, initializing [ 86.822671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 86.828099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 88.227922] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 93.279830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 93.313929] Lustre: 6470:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 93.324889] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 93.326489] Lustre: Skipped 1 previous similar message [ 93.363101] Lustre: lustre-MDT0001: new disk, initializing [ 93.384467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 93.391478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 93.396312] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 94.698777] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 96.886209] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 99.973437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 100.064825] Lustre: lustre-OST0000: new disk, initializing [ 100.066588] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 100.069126] Lustre: 8407:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 100.086818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 101.902511] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 102.442661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 102.446689] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 102.463337] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 107.591741] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 107.641644] Lustre: lustre-OST0001: new disk, initializing [ 107.644171] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 107.646991] Lustre: 9482:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 107.665746] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 109.903835] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 110.123884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 110.127742] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 110.140741] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 115.859762] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 123.595170] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 125.622387] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing check_logdir /tmp/testlogs/ [ 127.131437] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing yml_node [ 128.586465] Lustre: DEBUG MARKER: Client: 2.17.57.103 [ 129.379110] Lustre: DEBUG MARKER: MDS: 2.17.57.103 [ 130.201792] Lustre: DEBUG MARKER: OSS: 2.17.57.103 [ 130.778401] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 12 21:41:28 EDT 2026 [ 136.189538] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 136.677737] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 149.983499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 149.986765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 149.992392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 151.007433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 151.007924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 151.009476] Lustre: Skipped 1 previous similar message [ 151.014345] Lustre: Skipped 1 previous similar message [ 154.457630] Lustre: server umount lustre-MDT0000 complete [ 155.704749] LustreError: 6463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789263714 with bad export cookie 4968391520866785596 [ 155.706074] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 155.707873] LustreError: 6463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 156.127622] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 156.127826] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 156.131201] Lustre: Skipped 2 previous similar messages [ 156.133118] Lustre: Skipped 2 previous similar messages [ 160.159555] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 160.163365] Lustre: Skipped 1 previous similar message [ 161.887993] Lustre: server umount lustre-MDT0001 complete [ 169.275201] Lustre: server umount lustre-OST0000 complete [ 176.655035] Lustre: server umount lustre-OST0001 complete [ 178.732958] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid [ 181.678779] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 193.700849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.788118] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 193.799281] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 193.837882] Lustre: lustre-MDT0000: new disk, initializing [ 193.863568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.871142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 195.315562] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 199.814918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 199.859724] Lustre: 15020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 199.974249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 201.422786] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 205.367573] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 207.267484] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 209.579571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.700904] Lustre: lustre-OST0000: new disk, initializing [ 209.702361] Lustre: Skipped 1 previous similar message [ 209.704061] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 209.706216] Lustre: Skipped 2 previous similar messages [ 209.708923] Lustre: 16989:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 209.741462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.744831] Lustre: Skipped 1 previous similar message [ 211.199323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 211.202339] Lustre: Skipped 1 previous similar message [ 211.204404] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 211.242440] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 211.673521] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 215.305183] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 216.761263] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 216.845783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 218.270444] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 218.356872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 221.665396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 221.669056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 221.670547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 221.674517] Lustre: Skipped 1 previous similar message [ 225.107356] Lustre: server umount lustre-OST0000 complete [ 226.784985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 226.784985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 231.904728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 231.907071] Lustre: Skipped 4 previous similar messages [ 232.668053] Lustre: server umount lustre-MDT0000 complete [ 233.992557] LustreError: 15057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789263792 with bad export cookie 4968391520866787115 [ 233.997141] LustreError: 15057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 233.997624] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 234.109684] Lustre: server umount lustre-MDT0001 complete [ 236.213471] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 21:43:13 (1789263793) [ 242.718229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 242.889218] LustreError: 19856:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 242.897265] LustreError: 19856:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 242.914667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 244.273953] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 247.152367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 248.668909] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 251.140128] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 252.942686] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 253.661657] Lustre: Failing over lustre-MDT0000 [ 253.750506] Lustre: server umount lustre-MDT0000 complete [ 256.736642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 256.737347] LustreError: 19851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 256.741806] Lustre: Skipped 1 previous similar message [ 262.112367] LustreError: 20583:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 262.119612] LustreError: 20583:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 267.148886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 267.205306] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 267.232660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 267.235812] Lustre: Skipped 1 previous similar message [ 268.876100] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 272.352524] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 272.354654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 272.362185] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 272.854331] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 277.472162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 277.475786] LustreError: Skipped 1 previous similar message [ 277.477854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 289.247111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 289.315946] Lustre: server umount lustre-MDT0000 complete [ 290.652187] LustreError: 19838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789263849 with bad export cookie 4968391520866788529 [ 290.655248] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 290.657272] LustreError: 19838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 290.782600] Lustre: server umount lustre-MDT0001 complete [ 293.745816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 293.926856] LustreError: 23189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 293.956854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 293.960814] Lustre: Skipped 2 previous similar messages [ 295.457587] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 298.467308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 300.059746] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 302.460605] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 304.257988] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 304.980116] Lustre: Failing over lustre-MDT0000 [ 305.069617] Lustre: server umount lustre-MDT0000 complete [ 309.089531] LustreError: 23916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 309.096449] LustreError: 23916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 318.167419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 318.211173] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 319.748448] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 323.488583] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 323.553122] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 323.555201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 323.557277] Lustre: Skipped 1 previous similar message [ 323.561951] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 328.671962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 328.674618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 328.678724] Lustre: Skipped 3 previous similar messages [ 328.680489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 328.682253] Lustre: Skipped 7 previous similar messages [ 331.539809] Lustre: server umount lustre-MDT0000 complete [ 332.776532] LustreError: 23169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789263891 with bad export cookie 4968391520866789467 [ 332.779522] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 332.780757] LustreError: 23169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 332.895369] Lustre: server umount lustre-MDT0001 complete [ 335.619812] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 21:44:53 (1789263893) [ 341.160578] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid [ 344.679492] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 363.298451] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 374.124331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 374.694224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 374.753381] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 375.000610] Lustre: lustre-MDT0000: new disk, initializing [ 375.148064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 375.153537] Lustre: Skipped 2 previous similar messages [ 375.174614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 378.167016] hrtimer: interrupt took 2249955 ns [ 382.316565] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 397.941619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 398.103723] Lustre: 29554:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 398.683500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 404.601995] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 414.643048] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 419.055724] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 423.816913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 423.980585] Lustre: lustre-OST0000: new disk, initializing [ 423.982477] Lustre: Skipped 1 previous similar message [ 423.984736] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 423.987925] Lustre: Skipped 2 previous similar messages [ 423.993930] Lustre: 31523:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 425.539364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 425.546059] Lustre: Skipped 1 previous similar message [ 425.560965] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 425.593652] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 425.617951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 425.638158] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 427.795299] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 433.836781] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 438.809073] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 451.558527] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 451.560407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 451.564789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 451.564798] Lustre: Skipped 2 previous similar messages [ 451.565270] LustreError: Skipped 1 previous similar message [ 451.588687] Lustre: Skipped 2 previous similar messages [ 451.802817] Lustre: server umount lustre-OST0000 complete [ 460.079115] Lustre: server umount lustre-MDT0000 complete [ 461.793250] LustreError: 30395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 461.804378] LustreError: 30395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 462.149092] LustreError: 29546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789264020 with bad export cookie 4968391520866789873 [ 462.156752] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 462.158777] LustreError: 29546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 462.429908] Lustre: server umount lustre-MDT0001 complete [ 468.325246] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [ 469.947350] Key type lgssc unregistered [ 470.145990] LNet: 33497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 470.151410] LNetError: 33497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 470.165665] LNet: Removed LNI 192.168.204.110@tcp [ 470.658219] Key type .llcrypt unregistered [ 470.660204] Key type ._llcrypt unregistered [ 484.536289] Key type ._llcrypt registered [ 484.537947] Key type .llcrypt registered [ 484.611335] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 485.203767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 485.216779] alg: No test for adler32 (adler32-zlib) [ 486.136783] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 486.278069] LNet: Added LNI 192.168.204.110@tcp [8/256/0/180] [ 487.895163] Key type lgssc registered [ 488.611715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 493.162528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 493.190844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.454582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 497.065230] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 501.794328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 501.978300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 504.191567] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 508.321437] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 511.650771] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 515.245417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 515.461094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 518.495533] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 522.706493] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 526.774119] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 535.544933] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 21:48:12 (1789264092) [ 536.630816] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 537.859558] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 541.796969] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 21:48:19 (1789264099) [ 547.262633] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 21:48:24 (1789264104) [ 547.933361] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 553.618576] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 553.623111] Lustre: Skipped 2 previous similar messages [ 554.979971] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 21:48:32 (1789264112) [ 560.588918] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 560.595208] Lustre: Skipped 4 previous similar messages [ 568.941660] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 568.947153] Lustre: Skipped 5 previous similar messages [ 570.675632] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 21:48:47 (1789264127) [ 572.303930] Lustre: *** cfs_fail_loc=131b, val=2*** [ 574.535179] Lustre: *** cfs_fail_loc=131b, val=2*** [ 579.132339] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 21:48:56 (1789264136) [ 586.152977] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 586.158086] Lustre: Skipped 7 previous similar messages [ 587.194157] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 21:49:04 (1789264144) [ 595.424923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 595.432978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 595.444345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 598.499209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 598.501141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 598.522930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 601.197842] Lustre: server umount lustre-OST0000 complete [ 603.617118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.621076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 603.623672] Lustre: Skipped 1 previous similar message [ 603.635022] Lustre: Skipped 2 previous similar messages [ 608.736983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 608.745626] Lustre: Skipped 1 previous similar message [ 609.650487] Lustre: server umount lustre-MDT0000 complete [ 611.675632] LustreError: 38933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789264170 with bad export cookie 2626461204934132212 [ 611.678568] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 611.681711] LustreError: 38933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 611.838726] Lustre: server umount lustre-MDT0001 complete [ 617.530613] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [ 619.082693] Key type lgssc unregistered [ 619.253794] LNet: 42713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 619.261123] LNetError: 42713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 619.273253] LNet: Removed LNI 192.168.204.110@tcp [ 619.759599] Key type .llcrypt unregistered [ 619.763435] Key type ._llcrypt unregistered [ 630.068717] Key type ._llcrypt registered [ 630.073535] Key type .llcrypt registered [ 630.145790] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid [ 639.798085] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 640.355652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 640.409437] alg: No test for adler32 (adler32-zlib) [ 641.337124] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 641.489891] LNet: Added LNI 192.168.204.110@tcp [8/256/0/180] [ 643.119184] Key type lgssc registered [ 643.846416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 664.415572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 664.434256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 665.560327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 665.582230] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 665.633348] Lustre: lustre-MDT0000: new disk, initializing [ 665.684862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 665.700278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 667.958082] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 674.073095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 674.139637] Lustre: 45841:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 674.193886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 674.196724] Lustre: Skipped 1 previous similar message [ 674.245936] Lustre: lustre-MDT0001: new disk, initializing [ 674.314249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 674.340199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 674.349430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 676.568631] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 682.512963] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 685.683579] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 689.474626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 689.624620] Lustre: lustre-OST0000: new disk, initializing [ 689.628317] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 689.633917] Lustre: 47807:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 689.684053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 692.643456] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 697.891489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 697.900850] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 697.964916] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 698.639355] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 701.204856] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 701.337095] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 703.853639] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 703.957274] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 708.067332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 708.068352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 708.069289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 708.074618] Lustre: Skipped 1 previous similar message [ 713.190709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 713.193796] Lustre: Skipped 2 previous similar messages [ 718.307543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 718.313751] Lustre: Skipped 1 previous similar message [ 719.327311] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 719.454435] Lustre: server umount lustre-OST0000 complete [ 723.425653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 723.431994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 723.455179] Lustre: Skipped 1 previous similar message [ 723.467918] Lustre: Skipped 1 previous similar message [ 728.466988] Lustre: server umount lustre-MDT0000 complete [ 730.968651] LustreError: 46634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789264289 with bad export cookie 2073904357075973561 [ 730.972404] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 730.976446] LustreError: 46634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 731.212394] Lustre: server umount lustre-MDT0001 complete [ 737.436766] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 21:51:34 (1789264294) [ 741.505413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 741.794350] LustreError: 50265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 741.811578] LustreError: 50265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 741.851799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 744.389798] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 746.975960] LustreError: 50266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 749.201725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 749.394788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 751.525498] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 755.543084] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 758.724131] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 762.837208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 763.025523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 766.114514] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 769.891745] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 777.174088] LustreError: 52793:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 777.639340] Lustre: Modifying parameter general.timeout=1 in log params [ 778.665872] Lustre: Modifying parameter general.timeout=3 in log params [ 778.671285] Lustre: Skipped 1 previous similar message [ 781.125283] Lustre: Modifying parameter general.timeout=8 in log params [ 781.128857] Lustre: Skipped 4 previous similar messages [ 785.631217] Lustre: Modifying parameter general.timeout=17 in log params [ 785.636140] Lustre: Skipped 8 previous similar messages [ 793.681280] Lustre: Modifying parameter general.timeout=34 in log params [ 793.684685] Lustre: Skipped 16 previous similar messages [ 809.706552] Lustre: Modifying parameter general.timeout=69 in log params [ 809.709636] Lustre: Skipped 34 previous similar messages [ 842.724744] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 21:53:20 (1789264400) [ 843.315128] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 843.319927] Lustre: Skipped 61 previous similar messages [ 844.544792] Lustre: 59706:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 845.611393] Lustre: 59804:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 847.713300] Lustre: 60000:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 847.724274] Lustre: 60000:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 864.804492] Lustre: 61540:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 864.809968] Lustre: 61540:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 864.820591] LustreError: 61540:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 867.042458] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 21:53:44 (1789264424) [ 874.977755] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 874.980549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 874.991682] Lustre: Skipped 2 previous similar messages [ 875.000417] Lustre: Skipped 2 previous similar messages [ 875.248536] Lustre: server umount lustre-MDT0000 complete [ 877.034403] LustreError: 52843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789264435 with bad export cookie 2073904357075974436 [ 877.036391] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 877.042242] LustreError: 52843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 879.188617] Lustre: server umount lustre-MDT0001 complete [ 891.527497] Lustre: server umount lustre-OST0000 complete [ 905.633399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 905.797494] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 905.965573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 907.929713] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 911.945840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 912.015655] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 912.030529] Lustre: 64066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 914.068257] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 917.448586] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 919.901617] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 923.165052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 923.270523] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 926.267368] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 930.080723] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 930.891065] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 930.896252] Lustre: Skipped 22 previous similar messages [ 930.916608] Lustre: 66629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 934.370813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 934.372349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 934.373170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 934.378769] LustreError: Skipped 2 previous similar messages [ 934.388557] Lustre: Skipped 1 previous similar message [ 940.004072] Lustre: server umount lustre-OST0000 complete [ 943.072886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 943.076507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 954.848370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 954.850923] Lustre: Skipped 9 previous similar messages [ 956.384365] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 956.494941] Lustre: server umount lustre-MDT0000 complete [ 958.218498] LustreError: 64058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789264516 with bad export cookie 2073904357075976956 [ 958.221846] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 958.224860] LustreError: 64058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 958.359258] Lustre: server umount lustre-MDT0001 complete [ 962.969927] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [ 964.248388] Key type lgssc unregistered [ 964.414766] LNet: 67836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.419300] LNetError: 67836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 964.433764] LNet: Removed LNI 192.168.204.110@tcp [ 964.896293] Key type .llcrypt unregistered [ 964.897661] Key type ._llcrypt unregistered [ 976.758113] Key type ._llcrypt registered [ 976.759890] Key type .llcrypt registered [ 976.811879] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 977.379577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 977.389259] alg: No test for adler32 (adler32-zlib) [ 978.282159] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 978.397036] LNet: Added LNI 192.168.204.110@tcp [8/256/0/180] [ 979.999152] Key type lgssc registered [ 980.655218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 984.636380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 984.646071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 985.911831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 987.914818] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 992.198102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 992.551240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 994.713662] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 998.153487] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1000.817031] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1003.946298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1004.184304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1007.178910] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 1010.814357] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1020.387929] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1027.040832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1027.044839] LustreError: 69219:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1027.049069] Lustre: Skipped 2 previous similar messages [ 1027.060054] LustreError: 69219:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1027.083345] Lustre: server umount lustre-MDT0000 complete [ 1028.860568] LustreError: 69204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789264587 with bad export cookie 16406544639526017170 [ 1028.861626] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1028.869566] LustreError: 69204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1031.064977] Lustre: server umount lustre-MDT0001 complete [ 1049.635622] Lustre: server umount lustre-OST0000 complete [ 1063.748320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1063.935351] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1063.954811] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1064.089962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1066.022630] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 1070.162741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1070.226918] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1070.245709] Lustre: 74088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1070.470323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1072.487736] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 1076.033472] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1078.612404] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1081.933891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1082.044724] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1082.259545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1085.249653] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 1089.058668] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1090.000610] Lustre: 76663:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1091.012247] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1093.602406] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1093.604259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1093.607252] LustreError: Skipped 1 previous similar message [ 1093.617634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1095.340780] Lustre: server umount lustre-OST0000 complete [ 1097.283358] Lustre: server umount lustre-MDT0000 complete [ 1098.720530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1098.721226] LustreError: 74096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1098.727469] Lustre: Skipped 2 previous similar messages [ 1098.737326] LustreError: 74096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1103.841123] LustreError: 74097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1103.851065] LustreError: 74097:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1108.960606] LustreError: 75135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1108.967323] LustreError: 75135:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1114.079380] Lustre: 74092:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789264656/real 1789264656] req@00000000ca1bbc25 x1876179850165248/t0(0) o101->MGC192.168.204.110@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1789264672 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 [ 1114.081662] LustreError: 75133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1114.094229] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1114.104040] LustreError: 75133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1114.117478] LustreError: 74092:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 1114.259789] Lustre: server umount lustre-MDT0001 complete [ 1119.641256] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [ 1121.182845] Key type lgssc unregistered [ 1121.376064] LNet: 77863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1121.379722] LNetError: 77863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1121.390709] LNet: Removed LNI 192.168.204.110@tcp [ 1121.844839] Key type .llcrypt unregistered [ 1121.851097] Key type ._llcrypt unregistered [ 1134.429625] Key type ._llcrypt registered [ 1134.430865] Key type .llcrypt registered [ 1134.483572] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 1134.878270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1134.917076] alg: No test for adler32 (adler32-zlib) [ 1135.822918] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 1135.941661] LNet: Added LNI 192.168.204.110@tcp [8/256/0/180] [ 1137.543351] Key type lgssc registered [ 1138.075180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1141.843580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1141.852985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1143.100614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1144.977071] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 1149.750705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1150.088146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1152.137861] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 1155.619443] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1158.091410] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1161.171414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1161.446308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1164.279528] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 1167.802797] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1176.497154] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1177.399929] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1177.403922] Lustre: Skipped 1 previous similar message [ 1178.708647] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1178.712889] Lustre: Skipped 2 previous similar messages [ 1180.746502] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1180.749609] Lustre: Skipped 4 previous similar messages [ 1185.130819] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1185.133882] Lustre: Skipped 10 previous similar messages [ 1193.475917] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1193.479756] Lustre: Skipped 20 previous similar messages [ 1209.800643] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1209.803891] Lustre: Skipped 36 previous similar messages [ 1221.904445] Lustre: 87317:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1223.159718] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 21:59:40 (1789264780) [ 1223.950732] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1224.818448] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 21:59:42 (1789264782) [ 1229.280232] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1229.284141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1229.290390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1231.330259] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1231.331423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1231.333728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1232.415830] Lustre: server umount lustre-OST0000 complete [ 1236.449596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1236.460400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1236.464350] Lustre: Skipped 2 previous similar messages [ 1240.470731] Lustre: server umount lustre-MDT0000 complete [ 1241.568387] LustreError: 79249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1241.578485] LustreError: 79249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1242.445813] LustreError: 79229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789264800 with bad export cookie 7199860077628080586 [ 1242.447520] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1242.453486] LustreError: 79229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1242.604228] Lustre: server umount lustre-MDT0001 complete [ 1247.419965] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [ 1248.724628] Key type lgssc unregistered [ 1248.878225] LNet: 88562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1248.882355] LNetError: 88562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1248.892532] LNet: Removed LNI 192.168.204.110@tcp [ 1249.293875] Key type .llcrypt unregistered [ 1249.295673] Key type ._llcrypt unregistered [ 1251.164481] Key type ._llcrypt registered [ 1251.166286] Key type .llcrypt registered [ 1251.218252] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1253.266659] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules [ 1253.810510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1253.820189] alg: No test for adler32 (adler32-zlib) [ 1254.720980] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 1254.848651] LNet: Added LNI 192.168.204.110@tcp [8/256/0/180] [ 1256.455376] Key type lgssc registered [ 1257.011560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1257.540250] LustreError: 89711:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 1266.947071] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 1271.234371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1278.673504] LustreError: 89711:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1278.690249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1279.971732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1281.774449] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 1285.860181] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 22:00:43 (1789264843) [ 1288.090774] LustreError: 90769:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1288.096870] LustreError: 90769:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1288.104041] LustreError: 90769:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 1288.266291] Lustre: server umount lustre-MDT0000 complete [ 1294.782447] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [ 1296.078786] Key type lgssc unregistered [ 1296.238634] LNet: 92258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1296.243620] LNetError: 92258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1296.258834] LNet: Removed LNI 192.168.204.110@tcp [ 1296.681503] Key type .llcrypt unregistered [ 1296.683950] Key type ._llcrypt unregistered [ 1306.596653] Key type ._llcrypt registered [ 1306.598366] Key type .llcrypt registered [ 1306.657125] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 1307.209217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1307.218544] alg: No test for adler32 (adler32-zlib) [ 1308.110045] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 1308.225412] LNet: Added LNI 192.168.204.110@tcp [8/256/0/180] [ 1309.832644] Key type lgssc registered [ 1310.344745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1313.872944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1313.884570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1315.112353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1316.872874] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 1320.907126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1321.259358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1323.388343] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 1327.108541] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1329.684843] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1333.037233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1333.392786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1336.229248] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 1339.652691] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1377.013631] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 22:02:14 (1789264934) [ 1377.832803] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1378.722836] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 22:02:16 (1789264936) [ 1382.882418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1382.882907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1382.888874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1382.893088] Lustre: Skipped 2 previous similar messages [ 1382.895787] Lustre: Skipped 1 previous similar message [ 1386.538190] Lustre: server umount lustre-MDT0000 complete [ 1388.010591] LustreError: 93459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1388.024235] LustreError: 93459:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1388.751642] LustreError: 93444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789264947 with bad export cookie 11427667418238585320 [ 1388.769777] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1388.978770] Lustre: server umount lustre-MDT0001 complete [ 1402.293046] Lustre: server umount lustre-OST0000 complete [ 1419.435660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1420.035359] LustreError: 97755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1420.287386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1425.393951] LustreError: 97756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1428.075679] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 1429.551876] LustreError: 97755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1434.617474] LustreError: 97756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1443.033347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1443.729841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1449.728047] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 1455.994985] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1460.806778] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1471.428836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1471.613669] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1471.618606] LustreError: 97743:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1471.642496] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1471.654127] LustreError: 99841:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 1471.664874] LustreError: 99841:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 1471.671467] LustreError: 99841:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1471.750579] Lustre: server umount lustre-OST0000 complete [ 1471.754235] LustreError: 99841:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1478.727680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1478.894354] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1478.909413] LustreError: 97745:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1478.916819] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1478.926847] LustreError: 100303:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 1478.933187] LustreError: 100303:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 1478.938440] LustreError: 100303:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1478.991815] Lustre: server umount lustre-OST0000 complete [ 1478.995888] LustreError: 100303:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1491.067407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1491.503401] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1491.524777] Lustre: Found index 0 for lustre-OST0000, updating log [ 1491.555721] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1491.769178] Lustre: lustre-OST0000: new disk, initializing [ 1491.786073] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1491.803706] Lustre: 100837:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1492.278878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1502.678820] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 1510.338334] Lustre: Failing over lustre-OST0000 [ 1510.850811] Lustre: server umount lustre-OST0000 complete [ 1520.098000] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1520.105869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1520.124616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1522.514542] Lustre: server umount lustre-MDT0000 complete [ 1525.728425] LustreError: 97755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1525.747739] LustreError: 97755:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1527.162912] LustreError: 97736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789265085 with bad export cookie 11427667418238588057 [ 1527.165577] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1527.180430] LustreError: 97736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1527.541819] Lustre: server umount lustre-MDT0001 complete [ 1537.662549] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 22:04:54 (1789265094) [ 1553.697867] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 1567.290036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1567.661084] LustreError: 103184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1567.895132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1574.082979] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 1583.359160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1583.586691] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1583.593956] Lustre: Skipped 1 previous similar message [ 1589.411131] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 1592.411114] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1599.717240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1600.595413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1600.605719] Lustre: Skipped 1 previous similar message [ 1607.126485] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 1615.874027] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1615.942636] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1616.020211] Lustre: lustre-OST0001: new disk, initializing [ 1616.028870] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1616.044108] Lustre: 105503:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1623.286833] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 1625.655473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1625.663471] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1625.754440] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1631.703597] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1637.791906] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1645.537058] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1645.551049] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1645.572312] Lustre: Skipped 1 previous similar message [ 1645.583905] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1651.683769] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1651.699321] Lustre: Skipped 6 previous similar messages [ 1656.287779] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1656.474319] Lustre: server umount lustre-MDT0001 complete [ 1656.801112] LustreError: 103185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1656.832261] LustreError: 103185:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1666.007450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1666.144358] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1666.151382] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1666.205325] Lustre: 103171:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1666.955902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1666.964757] Lustre: Skipped 1 previous similar message [ 1667.775242] LustreError: 107053:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1667.783844] LustreError: 107053:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1667.795411] LustreError: 103176:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1667.905994] LustreError: 107063:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.110@tcp: cfg command failed: rc = -17 [ 1667.919799] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1667.919799] [ 1670.963468] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 1672.182716] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1680.974228] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 22:07:17 (1789265237) [ 1699.881943] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 1720.261446] Lustre: 109119:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1744.439335] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1748.468368] Lustre: 110254:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1840.607993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1840.614604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1840.626607] Lustre: Skipped 2 previous similar messages [ 1840.642614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1840.662478] Lustre: Skipped 1 previous similar message [ 1845.377396] Lustre: server umount lustre-MDT0000 complete [ 1846.244642] LustreError: 103181:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1846.267530] LustreError: 103181:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 1848.850129] LustreError: 103165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789265407 with bad export cookie 11427667418238588988 [ 1848.863320] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1849.168421] Lustre: server umount lustre-MDT0001 complete [ 1863.616653] Lustre: server umount lustre-OST0000 complete [ 1866.534139] Lustre: 92774:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789265409/real 1789265409] req@ffff8d52ce7d1c00 x1876180196478592/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789265425 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1866.559513] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1866.572550] Lustre: Skipped 3 previous similar messages [ 1867.673088] Lustre: server umount lustre-OST0001 complete [ 1873.751046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1884.026557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1891.898681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1903.810730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1926.882784] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 1937.401833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1937.416392] Lustre: lustre-MDT0000: reset Object Index mappings [ 1938.070733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1942.301322] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 1949.548921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1949.563583] Lustre: lustre-MDT0001: reset Object Index mappings [ 1953.909674] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 1956.579911] Lustre: 115362:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1963.152878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1969.143465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 1969.917693] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 1974.249859] LustreError: 115715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1974.270656] LustreError: 115715:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 1975.283240] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 1978.137578] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1984.000946] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 1984.004803] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 1984.729239] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 1992.866978] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1996.576559] Lustre: 117239:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2055.499233] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 22:13:32 (1789265612) [ 2081.251506] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2081.267538] Lustre: Skipped 1 previous similar message [ 2081.269699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2081.279018] Lustre: Skipped 4 previous similar messages [ 2087.094213] Lustre: server umount lustre-MDT0000 complete [ 2091.104877] LustreError: 114194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789265649 with bad export cookie 11427667418238635874 [ 2091.113103] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2091.123242] LustreError: 114194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2097.665367] Lustre: server umount lustre-MDT0001 complete [ 2101.750788] LustreError: 117994:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000401:0x139:0x0].0x0 (ffff8d52c1204000) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 2116.070050] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2116.167607] Lustre: server umount lustre-OST0000 complete [ 2126.275435] Lustre: server umount lustre-OST0001 complete [ 2133.846623] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid [ 2144.464679] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 2187.677561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2187.924758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2187.965335] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2188.092865] Lustre: lustre-MDT0000: new disk, initializing [ 2188.192821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2193.364759] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 2200.654740] Lustre: server umount lustre-MDT0000 complete [ 2210.934403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2211.241760] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2211.255201] Lustre: 121512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2211.272538] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2211.579727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2211.585618] Lustre: Skipped 4 previous similar messages [ 2216.261512] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 2221.280531] Lustre: server umount lustre-MDT0000 complete [ 2230.677680] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 22:16:27 (1789265787) [ 2232.363433] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2234.749552] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 22:16:31 (1789265791) [ 2255.569655] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 2266.706392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2272.738648] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 2282.339435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2282.447767] Lustre: 123563:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2282.505026] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2282.509159] Lustre: Skipped 1 previous similar message [ 2282.581369] Lustre: lustre-MDT0001: new disk, initializing [ 2282.740275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2282.763658] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2287.983782] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 2293.979605] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2301.147047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2301.418241] Lustre: lustre-OST0000: new disk, initializing [ 2301.426739] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2301.439868] Lustre: 125144:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2302.819943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2302.829314] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2302.960686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2307.847662] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 2318.261953] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2318.430985] Lustre: lustre-OST0001: new disk, initializing [ 2318.435538] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2318.443691] Lustre: 126016:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2319.934174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2319.944244] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2320.136367] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2324.934442] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 2334.954496] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2339.193275] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2757.759859] Lustre: 132435:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2757.777400] Lustre: 132435:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2757.784068] Lustre: 132435:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2757.793213] Lustre: 132435:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2757.803479] Lustre: 132435:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2757.810927] Lustre: 132435:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3071.014784] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:30:26 (1789266626) [ 3091.428975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3091.454296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3091.456763] Lustre: Skipped 7 previous similar messages [ 3091.471366] Lustre: Skipped 11 previous similar messages [ 3093.691447] Lustre: server umount lustre-MDT0000 complete [ 3096.561642] LustreError: 126827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3096.586371] LustreError: 126827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3097.611661] LustreError: 123555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789266656 with bad export cookie 11427667418238705993 [ 3097.630639] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3097.633340] LustreError: 123555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3101.667576] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3101.671492] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3101.679220] Lustre: Skipped 1 previous similar message [ 3101.706615] Lustre: Skipped 1 previous similar message [ 3104.191660] Lustre: server umount lustre-MDT0001 complete [ 3113.782684] Lustre: server umount lustre-OST0000 complete [ 3123.855671] Lustre: server umount lustre-OST0001 complete [ 3130.013840] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid [ 3139.191584] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 3174.622137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3174.951668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3175.019453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3175.152654] Lustre: lustre-MDT0000: new disk, initializing [ 3175.249970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3175.254295] Lustre: Skipped 4 previous similar messages [ 3175.284782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3179.822764] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 3190.125670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3190.257094] Lustre: 135499:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3190.313295] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3190.317767] Lustre: Skipped 1 previous similar message [ 3190.474272] Lustre: lustre-MDT0001: new disk, initializing [ 3190.634578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3190.644096] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3195.108810] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 3204.139807] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3209.886472] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3217.366720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3217.721887] Lustre: lustre-OST0000: new disk, initializing [ 3217.729946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3217.738376] Lustre: 137467:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3219.737929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3219.759945] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3219.879914] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3224.701187] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 3234.167619] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3241.978190] Lustre: lustre-MDD0000: changelog on [ 3244.293548] Lustre: lustre-MDD0001: changelog on [ 5504.998652] Lustre: lustre-MDD0001: changelog off [ 5507.483914] Lustre: 135506:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5508.712714] Lustre: lustre-MDD0000: changelog off [ 5516.770658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5516.776467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5516.802415] Lustre: Skipped 1 previous similar message [ 5516.819384] Lustre: Skipped 3 previous similar messages [ 5521.897364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5521.907991] Lustre: Skipped 1 previous similar message [ 5522.819090] Lustre: server umount lustre-OST0000 complete [ 5530.596907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5530.608671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5530.639573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5532.132255] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5534.279796] Lustre: server umount lustre-MDT0000 complete [ 5537.252479] LustreError: 135510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5537.274104] LustreError: 135510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5539.518947] LustreError: 136292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789269098 with bad export cookie 11427667418239529270 [ 5539.526449] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5539.532921] LustreError: 136292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5540.049895] Lustre: server umount lustre-MDT0001 complete [ 5553.687984] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [ 5557.733336] Key type lgssc unregistered [ 5558.210439] LNet: 146671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5558.217605] LNetError: 146671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5558.240496] LNet: Removed LNI 192.168.204.110@tcp [ 5559.336337] Key type .llcrypt unregistered [ 5559.338468] Key type ._llcrypt unregistered [ 5586.271995] Key type ._llcrypt registered [ 5586.282639] Key type .llcrypt registered [ 5586.464712] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid [ 5602.132777] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 5603.576290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5603.619524] alg: No test for adler32 (adler32-zlib) [ 5605.001777] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 5605.415237] LNet: Added LNI 192.168.204.110@tcp [8/256/0/180] [ 5607.240419] Key type lgssc registered [ 5608.786635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5653.590905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5653.629346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5655.032026] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5655.084606] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5655.194605] Lustre: lustre-MDT0000: new disk, initializing [ 5655.330111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5655.354534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5660.939349] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 5674.042137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5674.199377] Lustre: 149871:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5674.278785] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5674.291976] Lustre: Skipped 1 previous similar message [ 5674.439994] Lustre: lustre-MDT0001: new disk, initializing [ 5674.600501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5674.676031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5674.694848] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5679.571824] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 5690.907643] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5698.096073] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5705.681145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5706.082500] Lustre: lustre-OST0000: new disk, initializing [ 5706.087544] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5706.093226] Lustre: 151843:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5706.184315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5714.839863] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 5716.017451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5716.036546] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5716.223661] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5727.191425] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5734.037827] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_module obdecho/obdecho [ 5736.563417] LustreError: 152847:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5736.574984] LustreError: 152847:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5738.660396] LustreError: 152944:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5752.299971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5752.303597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5752.322541] Lustre: Skipped 2 previous similar messages [ 5752.338463] Lustre: Skipped 2 previous similar messages [ 5757.419414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5757.422027] Lustre: Skipped 2 previous similar messages [ 5757.844828] Lustre: server umount lustre-MDT0000 complete [ 5762.182830] LustreError: 149864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789269320 with bad export cookie 145798991054685366 [ 5762.187275] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5762.203330] LustreError: 149864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5762.535538] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5762.811178] Lustre: server umount lustre-MDT0001 complete [ 5769.705323] Lustre: server umount lustre-OST0000 complete [ 5794.372832] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid [ 5804.690799] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 5849.654062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5850.056613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5850.128750] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5850.271498] Lustre: lustre-MDT0000: new disk, initializing [ 5850.404390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5850.461685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5855.777742] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 5867.514633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5867.717479] Lustre: 156659:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5867.797133] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5867.804556] Lustre: Skipped 1 previous similar message [ 5867.923371] Lustre: lustre-MDT0001: new disk, initializing [ 5868.051177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5868.098728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5868.108209] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5873.875901] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 5884.745330] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5892.597311] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5902.876557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5903.273483] Lustre: lustre-OST0000: new disk, initializing [ 5903.277423] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5903.288347] Lustre: 158633:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5903.428335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5909.098302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5909.122238] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5909.300306] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5911.356546] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 5923.287706] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5928.412664] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5928.680464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5934.052803] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5934.365833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5940.193664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5940.198838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5940.203237] LustreError: Skipped 1 previous similar message [ 5940.220879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5940.226286] Lustre: Skipped 1 previous similar message [ 5942.827450] Lustre: server umount lustre-OST0000 complete [ 5949.923227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5949.947105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5949.961788] Lustre: Skipped 1 previous similar message [ 5949.977947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5953.716904] Lustre: server umount lustre-MDT0000 complete [ 5955.557433] LustreError: 156667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5955.584639] LustreError: 156667:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5958.869826] LustreError: 157465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789269517 with bad export cookie 145798991054686325 [ 5958.879053] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5958.892362] LustreError: 157465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5959.304202] Lustre: server umount lustre-MDT0001 complete [ 5962.992510] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 23:18:39 (1789269519) [ 5965.631441] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5967.908409] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 23:18:44 (1789269524) [ 5977.340060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5977.716867] LustreError: 161044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5977.796862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5982.657304] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 5983.203348] LustreError: 161045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5988.328802] LustreError: 161044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5992.822847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5998.348685] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 6007.172044] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6013.940449] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6020.956026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6021.435625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6021.455678] Lustre: Skipped 1 previous similar message [ 6029.496510] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 6038.340695] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6048.252390] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6048.274656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6048.281107] Lustre: Skipped 3 previous similar messages [ 6048.298964] Lustre: Skipped 4 previous similar messages [ 6051.444779] Lustre: server umount lustre-MDT0000 complete [ 6053.351514] LustreError: 161041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6053.380285] LustreError: 161041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6055.643396] LustreError: 161027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789269614 with bad export cookie 145798991054687207 [ 6055.652444] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6055.659456] LustreError: 161027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6058.465845] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6058.478897] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6058.483043] Lustre: Skipped 1 previous similar message [ 6062.165361] Lustre: server umount lustre-MDT0001 complete [ 6069.608357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6083.255368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6083.561805] LustreError: 164890:0:(llog.c:1655:llog_backup()) MGC192.168.204.110@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6083.574208] Lustre: 164890:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.110@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6086.114229] LustreError: 164895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6086.123482] LustreError: 164895:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6104.571364] LustreError: 164895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6104.614467] LustreError: 164895:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6136.801228] LustreError: 164895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6136.835851] LustreError: 164895:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 25 previous similar messages [ 6144.479304] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6144.485073] Lustre: 164890:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.110@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6185.485525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6190.566098] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6191.567895] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 6200.801248] LustreError: 164897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6200.845149] LustreError: 164897:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 41 previous similar messages [ 6202.590622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6216.166402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6226.408458] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x205fb68ee6ce540 [ 6226.420563] Lustre: MGC192.168.204.110@tcp: Connection restored to 0@lo (at 0@lo) [ 6232.062308] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6232.266152] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 6241.274890] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6243.563676] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6250.120523] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6251.875901] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6272.994423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6273.002956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6273.026659] Lustre: Skipped 1 previous similar message [ 6273.035110] Lustre: Skipped 2 previous similar messages [ 6273.036476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6274.655918] Lustre: server umount lustre-MDT0000 complete [ 6279.383451] LustreError: 161026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789269837 with bad export cookie 145798991054693696 [ 6279.388092] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6279.400859] LustreError: 161026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6280.225377] Lustre: server umount lustre-MDT0001 complete [ 6294.780619] Lustre: server umount lustre-OST0000 complete [ 6319.107195] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid [ 6329.256855] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 6376.867567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6377.053168] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6377.076258] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6377.171510] Lustre: lustre-MDT0000: new disk, initializing [ 6377.246268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6377.249596] Lustre: Skipped 1 previous similar message [ 6377.259388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6383.976839] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 6396.252382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6396.378277] Lustre: 170005:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6396.814253] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6402.561906] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 6412.844491] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6418.545096] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6426.583466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6426.926939] Lustre: lustre-OST0000: new disk, initializing [ 6426.930605] Lustre: Skipped 1 previous similar message [ 6426.940495] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6426.954691] Lustre: Skipped 2 previous similar messages [ 6426.958963] Lustre: 171976:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6434.620401] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 6435.996667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6436.015491] Lustre: Skipped 1 previous similar message [ 6436.034805] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6436.232586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6445.110628] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6451.051485] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6451.232824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6457.020513] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6457.312809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6461.923792] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6461.930022] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6461.943147] LustreError: Skipped 1 previous similar message [ 6461.944098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6461.983328] Lustre: Skipped 2 previous similar messages [ 6465.971849] Lustre: server umount lustre-OST0000 complete [ 6476.951862] Lustre: server umount lustre-MDT0000 complete [ 6477.281538] LustreError: 171174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6477.305522] LustreError: 171174:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 24 previous similar messages [ 6480.543324] LustreError: 169996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789270039 with bad export cookie 145798991054710503 [ 6480.549812] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6480.558129] LustreError: 169996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6480.972085] Lustre: server umount lustre-MDT0001 complete [ 6492.498642] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 23:27:28 (1789270048) [ 6503.551210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6511.414504] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 6518.982196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6524.366616] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 6531.851629] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6537.414563] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6545.500169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6552.376126] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 6560.111968] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6573.705765] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 23:28:50 (1789270130) [ 6579.684056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6579.693717] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6579.700372] Lustre: Skipped 2 previous similar messages [ 6579.702098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6579.702105] Lustre: Skipped 3 previous similar messages [ 6583.211704] Lustre: server umount lustre-OST0000 complete [ 6601.183407] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6601.279275] LustreError: 177365:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x70616d65646f6e:0x4:0x0].0x0 (000000000868e6f6) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 6601.397591] Lustre: server umount lustre-MDT0000 complete [ 6604.675696] LustreError: 174370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789270163 with bad export cookie 145798991054711385 [ 6604.687924] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6604.696108] LustreError: 174370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6605.121481] Lustre: server umount lustre-MDT0001 complete [ 6615.601147] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [ 6618.557528] Key type lgssc unregistered [ 6618.913177] LNet: 178117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6618.922776] LNetError: 178117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6618.941408] LNet: Removed LNI 192.168.204.110@tcp [ 6619.765374] Key type .llcrypt unregistered [ 6619.768727] Key type ._llcrypt unregistered [ 6639.915548] Key type ._llcrypt registered [ 6639.919448] Key type .llcrypt registered [ 6640.021604] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 6640.782959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6640.821262] alg: No test for adler32 (adler32-zlib) [ 6642.204896] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 6642.593840] LNet: Added LNI 192.168.204.110@tcp [8/256/0/180] [ 6644.335194] Key type lgssc registered [ 6645.716810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6654.474662] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6654.512122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6670.176299] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6675.295387] LustreError: 179357:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.110@tcp: failed processing log, type 4: rc = -110 [ 6680.415564] LustreError: 179357:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6680.431529] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6680.442435] LustreError: MGC192.168.204.110@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6680.462326] LustreError: 179357:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6680.485335] LustreError: 179357:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 6680.501649] LustreError: 179357:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 6680.595740] Lustre: server umount lustre-OST0000 complete [ 6680.599799] LustreError: 179357:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6691.066760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6692.594106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6697.395953] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 6703.078162] LustreError: 179857:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6703.110660] LustreError: 179857:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6707.018577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6707.501183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6712.863053] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 6720.608208] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6727.658276] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6736.345480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6736.584701] LustreError: 181673:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6736.599368] Lustre: 181673:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.110@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6736.842405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6743.720241] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 6752.361536] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6757.345054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6757.348890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6757.360169] Lustre: Skipped 2 previous similar messages [ 6757.367751] Lustre: Skipped 1 previous similar message [ 6760.764258] Lustre: server umount lustre-MDT0000 complete [ 6762.468279] LustreError: 179852:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6762.485500] LustreError: 179852:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6763.909839] LustreError: 179836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789270322 with bad export cookie 5781711865244640586 [ 6763.916232] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6763.921796] LustreError: 179836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6764.307654] Lustre: server umount lustre-MDT0001 complete [ 6778.316131] Lustre: server umount lustre-OST0000 complete [ 6789.810518] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid [ 6804.476554] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 6848.637817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6848.964273] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6848.989360] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6849.162679] Lustre: lustre-MDT0000: new disk, initializing [ 6849.351201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6849.398398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6854.953432] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 6866.878408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6867.008899] Lustre: 185212:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6867.055095] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6867.059822] Lustre: Skipped 1 previous similar message [ 6867.141719] Lustre: lustre-MDT0001: new disk, initializing [ 6867.226653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6867.266903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6867.281678] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6872.613471] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 6883.323593] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6890.374464] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6898.751857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6899.182924] Lustre: lustre-OST0000: new disk, initializing [ 6899.190042] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6899.202195] Lustre: 187181:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6899.332566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6905.432890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6905.445926] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6905.572822] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6907.365317] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 6918.425219] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6924.493732] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6924.800491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6931.542380] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6931.921936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6936.551233] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6936.561532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6936.565709] LustreError: Skipped 1 previous similar message [ 6936.570078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6936.588565] Lustre: Skipped 1 previous similar message [ 6936.608084] Lustre: Skipped 2 previous similar messages [ 6941.009184] Lustre: server umount lustre-OST0000 complete [ 6946.791520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6946.801740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6946.825422] Lustre: Skipped 1 previous similar message [ 6946.832506] Lustre: Skipped 1 previous similar message [ 6951.910206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6951.929753] Lustre: Skipped 1 previous similar message [ 6952.123660] Lustre: server umount lustre-MDT0000 complete [ 6956.317251] LustreError: 185206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789270514 with bad export cookie 5781711865244641314 [ 6956.330250] LustreError: 185206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6956.335157] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6956.625247] Lustre: server umount lustre-MDT0001 complete [ 6968.819368] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 23:35:24 (1789270524) [ 6992.549261] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [ 6995.498593] Key type lgssc unregistered [ 6995.794819] LNet: 190129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6995.811514] LNetError: 190129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6995.824749] LNet: Removed LNI 192.168.204.110@tcp [ 6996.956957] Key type .llcrypt unregistered [ 6996.959948] Key type ._llcrypt unregistered [ 7018.499427] Key type ._llcrypt registered [ 7018.507716] Key type .llcrypt registered [ 7018.647901] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 7020.002533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7020.095869] alg: No test for adler32 (adler32-zlib) [ 7021.220881] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 7021.598630] LNet: Added LNI 192.168.204.110@tcp [8/256/0/180] [ 7023.314191] Key type lgssc registered [ 7024.332844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7032.816134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7032.839044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7034.422061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7040.062902] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 7043.563718] LustreError: 191344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7043.593698] LustreError: 191344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7048.694063] LustreError: 191345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7050.714402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7051.273922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7056.680078] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 7066.775734] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7074.762197] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7083.877848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7084.065712] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7084.067763] LustreError: 193213:0:(llog.c:1696:llog_backup()) MGC192.168.204.110@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7084.078258] Lustre: 193213:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.110@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7084.179219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7090.788532] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 7101.083972] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7107.552619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7107.561201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7107.579134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7109.097406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7109.107101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7110.833497] Lustre: server umount lustre-MDT0000 complete [ 7114.212694] LustreError: 191345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7114.245516] LustreError: 191345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7115.990727] LustreError: 191324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789270674 with bad export cookie 1740912434242051778 [ 7116.006346] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7116.010511] LustreError: 191324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7116.586637] Lustre: server umount lustre-MDT0001 complete [ 7131.918894] Lustre: server umount lustre-OST0000 complete [ 7144.515934] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid [ 7159.099155] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 7203.433705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7203.649551] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7203.681278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7203.759716] Lustre: lustre-MDT0000: new disk, initializing [ 7203.888071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7203.907223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7209.403572] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 7221.007156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7221.102690] Lustre: 196741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7221.147623] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7221.153360] Lustre: Skipped 1 previous similar message [ 7221.156265] LustreError: 197523:0:(llog.c:1696:llog_backup()) MGC192.168.204.110@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7221.164217] LustreError: 197523:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7221.168848] Lustre: 197523:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.110@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7221.185580] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7221.193716] Lustre: Skipped 1 previous similar message [ 7221.259473] Lustre: lustre-MDT0001: new disk, initializing [ 7221.342392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7221.392763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7221.408620] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7226.312882] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 7237.360864] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7243.444589] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7251.463591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7251.739735] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7251.742674] Lustre: Skipped 1 previous similar message [ 7251.745620] LustreError: 198687:0:(llog.c:1696:llog_backup()) MGC192.168.204.110@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7251.754495] LustreError: 198687:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7251.759569] Lustre: 198687:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.110@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7251.766723] Lustre: 198687:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7251.787451] Lustre: lustre-OST0000: new disk, initializing [ 7251.791747] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7251.801245] Lustre: 198710:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7251.901235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7259.035260] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 7260.242936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7260.253207] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7260.424524] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7269.113295] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7273.423943] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7273.695523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7277.638688] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7277.795808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7281.122963] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7281.137286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7281.163940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7281.163952] Lustre: Skipped 2 previous similar messages [ 7281.177435] Lustre: Skipped 2 previous similar messages [ 7286.008031] Lustre: server umount lustre-OST0000 complete [ 7291.362705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7291.372182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7291.385353] Lustre: Skipped 1 previous similar message [ 7291.411828] Lustre: Skipped 2 previous similar messages [ 7296.482485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7296.508887] Lustre: Skipped 1 previous similar message [ 7305.183484] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7305.380373] Lustre: server umount lustre-MDT0000 complete [ 7305.698316] LustreError: 196748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7305.708580] LustreError: 196748:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7309.052557] LustreError: 197544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789270867 with bad export cookie 1740912434242052499 [ 7309.066523] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7309.078502] LustreError: 197544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7309.554090] Lustre: server umount lustre-MDT0001 complete [ 7320.464560] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 23:41:16 (1789270876) [ 7341.640829] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 7353.573257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7354.120528] LustreError: 201839:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7354.230730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7359.770893] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 7363.555653] LustreError: 201840:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7363.566968] LustreError: 201840:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7370.158607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7375.502514] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 7378.202269] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7385.593505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7396.196205] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 7405.693967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7405.848498] Lustre: lustre-OST0001: new disk, initializing [ 7405.855658] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7405.861876] Lustre: 204150:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7405.979813] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7405.984505] Lustre: Skipped 2 previous similar messages [ 7411.523851] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 7412.873145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7412.893514] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7413.075063] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7424.119146] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7432.905615] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7434.883453] Lustre: DEBUG MARKER: ADD OST3 [ 7441.567383] loop: module loaded [ 7441.681777] loop0: detected capacity change from 0 to 204800000 [ 7442.157133] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7442.176033] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7442.192727] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7449.943368] loop0: detected capacity change from 0 to 204800000 [ 7450.030338] Lustre: DEBUG MARKER: STOP OST3 [ 7456.848354] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7457.126886] Lustre: lustre-OST0003: new disk, initializing [ 7457.142262] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7457.160789] Lustre: 206196:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7458.468201] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7458.649214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7458.663081] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7458.729677] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7460.512071] LustreError: 206302:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7460.663885] Lustre: Failing over lustre-OST0003 [ 7460.967069] Lustre: server umount lustre-OST0003 complete [ 7463.932833] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7463.947394] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7463.956264] LustreError: 203699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7463.956275] LustreError: 203699:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7463.972908] Lustre: Skipped 1 previous similar message [ 7464.015062] LustreError: Skipped 1 previous similar message [ 7465.578690] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 7479.087613] Lustre: DEBUG MARKER: START OST3 again [ 7488.704230] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7489.150403] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7489.163596] Lustre: Skipped 1 previous similar message [ 7489.191330] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7489.539456] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7491.130442] Lustre: lustre-OST0003: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 7491.133893] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7495.035188] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 7517.499795] Lustre: Failing over lustre-OST0003 [ 7517.955156] Lustre: server umount lustre-OST0003 complete [ 7520.250867] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7520.275356] LustreError: 203699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7520.287993] LustreError: 203699:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 7539.682922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7539.693072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7539.722972] Lustre: Skipped 1 previous similar message [ 7539.734374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7539.740458] Lustre: Skipped 2 previous similar messages [ 7543.765278] Lustre: server umount lustre-MDT0000 complete [ 7547.839975] LustreError: 201821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271106 with bad export cookie 1740912434242053381 [ 7547.852420] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7547.859908] LustreError: 201821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7558.114879] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7558.118213] Lustre: Skipped 9 previous similar messages [ 7562.207128] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7562.541877] Lustre: server umount lustre-MDT0001 complete [ 7566.439430] Lustre: server umount lustre-OST0000 complete [ 7570.574398] Lustre: server umount lustre-OST0001 complete [ 7579.573475] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid [ 7590.834476] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 7632.854157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7633.130773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7633.172586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7633.316418] Lustre: lustre-MDT0000: new disk, initializing [ 7633.426435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7633.452683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7638.609233] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 7650.182139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7650.367317] Lustre: 211769:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7650.468511] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7650.476118] Lustre: Skipped 1 previous similar message [ 7650.612592] Lustre: lustre-MDT0001: new disk, initializing [ 7650.823692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7650.840889] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7655.983577] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 7666.567792] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7672.755988] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7679.744822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7679.964423] Lustre: 213737:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7681.351547] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7681.462961] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7686.331513] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 7696.711143] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7701.718650] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7701.947367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7707.268885] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7707.524906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7712.234340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7712.237312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7712.247499] Lustre: Skipped 6 previous similar messages [ 7712.248386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7712.257520] Lustre: Skipped 1 previous similar message [ 7715.793618] Lustre: server umount lustre-OST0000 complete [ 7722.476299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7726.655748] Lustre: server umount lustre-MDT0000 complete [ 7727.586733] LustreError: 211781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7727.601201] LustreError: 211781:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 7731.437067] LustreError: 215280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271289 with bad export cookie 1740912434242055670 [ 7731.438910] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7731.457850] LustreError: 215280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7731.831034] Lustre: server umount lustre-MDT0001 complete [ 7736.329716] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 23:48:12 (1789271292) [ 7753.348290] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid [ 7767.050532] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 7808.547779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7808.727318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7808.746935] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7808.750556] Lustre: Skipped 1 previous similar message [ 7808.886809] Lustre: lustre-MDT0000: new disk, initializing [ 7808.888767] Lustre: Skipped 1 previous similar message [ 7808.960204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7808.964601] Lustre: Skipped 1 previous similar message [ 7813.674511] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 7824.632843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7824.830517] Lustre: 218351:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7825.193583] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7830.366904] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 7840.832878] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7847.477430] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7854.810815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7855.055864] Lustre: 220322:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7856.394557] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7862.121596] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 7872.410354] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7878.163740] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7878.461063] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7883.092825] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7883.311954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7887.329656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7887.330696] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7887.344289] LustreError: Skipped 1 previous similar message [ 7887.368307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7887.375279] Lustre: Skipped 3 previous similar messages [ 7887.376988] Lustre: Skipped 2 previous similar messages [ 7891.629811] Lustre: server umount lustre-OST0000 complete [ 7910.367219] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7910.702452] Lustre: server umount lustre-MDT0000 complete [ 7911.905623] LustreError: 218362:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7911.919566] LustreError: 218362:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7915.103385] LustreError: 218343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271473 with bad export cookie 1740912434242056552 [ 7915.114773] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7915.134470] LustreError: 218343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7915.624750] Lustre: server umount lustre-MDT0001 complete [ 7925.632261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7926.137419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7926.141435] Lustre: Skipped 5 previous similar messages [ 7931.700617] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 7941.970696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7947.795487] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 7956.123666] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7962.403852] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7970.740267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7978.291076] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 7985.464839] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7997.422092] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7997.433492] LustreError: Skipped 1 previous similar message [ 8001.778893] Lustre: server umount lustre-OST0000 complete [ 8012.483890] Lustre: server umount lustre-MDT0000 complete [ 8017.181187] LustreError: 222519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271575 with bad export cookie 1740912434242057434 [ 8017.191627] LustreError: 222519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8017.197713] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8017.627604] Lustre: server umount lustre-MDT0001 complete [ 8126.111364] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [ 8129.315890] Key type lgssc unregistered [ 8129.576932] LNet: 226201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8129.588903] LNetError: 226201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8129.606161] LNet: Removed LNI 192.168.204.110@tcp [ 8130.409242] Key type .llcrypt unregistered [ 8130.413502] Key type ._llcrypt unregistered [ 8161.032386] Key type ._llcrypt registered [ 8161.039786] Key type .llcrypt registered [ 8161.205671] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid [ 8176.777910] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 8178.054380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8178.192114] alg: No test for adler32 (adler32-zlib) [ 8179.351577] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 8179.624909] LNet: Added LNI 192.168.204.110@tcp [8/256/0/180] [ 8181.333138] Key type lgssc registered [ 8182.701720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8226.549932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8226.581407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8227.886584] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8227.959139] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8228.096478] Lustre: lustre-MDT0000: new disk, initializing [ 8228.170182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8228.188990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8233.703956] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 8246.251423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8246.385900] Lustre: 229551:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8246.437428] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8246.441417] Lustre: Skipped 1 previous similar message [ 8246.514376] Lustre: lustre-MDT0001: new disk, initializing [ 8246.615350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8246.656151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8246.672741] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8252.452715] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 8264.264943] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8270.506451] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8278.196762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8278.589390] Lustre: lustre-OST0000: new disk, initializing [ 8278.594744] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8278.605818] Lustre: 231524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8278.688767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8285.721512] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 8288.323228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8288.335207] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8288.462066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8296.606407] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8301.706144] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8301.966659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8308.202341] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8308.581076] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8314.336419] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8314.337734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8314.353718] LustreError: Skipped 1 previous similar message [ 8314.378855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8314.386687] Lustre: Skipped 1 previous similar message [ 8317.379853] Lustre: server umount lustre-OST0000 complete [ 8323.552757] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8323.561829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8323.584098] Lustre: Skipped 1 previous similar message [ 8323.591336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8324.584036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8324.603771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8324.616415] Lustre: Skipped 1 previous similar message [ 8328.220771] Lustre: server umount lustre-MDT0000 complete [ 8332.910803] LustreError: 229544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271891 with bad export cookie 3175348121691663159 [ 8332.913122] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8332.928642] LustreError: 229544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8333.503896] Lustre: server umount lustre-MDT0001 complete [ 8354.396581] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 8365.958803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8366.290574] LustreError: 234458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8366.305764] LustreError: 234458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8366.339693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8371.038939] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 8371.688294] LustreError: 234459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8375.782960] LustreError: 234458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8380.081441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8380.359917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8384.984878] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 8387.750061] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8394.405377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8394.753389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8401.633863] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 8411.712937] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8411.953280] Lustre: lustre-OST0001: new disk, initializing [ 8411.957875] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8411.968063] Lustre: 236768:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8412.094782] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8419.942783] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 8420.998841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8421.010727] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8421.151111] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8431.839355] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8438.446274] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8451.697863] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 00:00:08 (1789272008) [ 8457.189846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8457.198084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8457.205807] Lustre: Skipped 3 previous similar messages [ 8457.227677] Lustre: Skipped 3 previous similar messages [ 8460.198188] Lustre: server umount lustre-MDT0000 complete [ 8462.350632] LustreError: 234458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8462.391471] LustreError: 234458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8464.348311] LustreError: 234439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272022 with bad export cookie 3175348121691664041 [ 8464.350651] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8464.360091] LustreError: 234439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8464.780589] Lustre: server umount lustre-MDT0001 complete [ 8479.260796] Lustre: server umount lustre-OST0000 complete [ 8483.273878] Lustre: 227776:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789272025/real 1789272025] req@ffff8d53ff237b80 x1876187402059904/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789272041 ref 1 fl Rpc:EXNQr/200/ffffffff rc -5/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8492.685562] Lustre: server umount lustre-OST0001 complete [ 8500.175648] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid [ 8511.612940] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 8549.985706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8550.228527] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8550.263452] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8550.392515] Lustre: lustre-MDT0000: new disk, initializing [ 8550.474547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8550.491469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8556.187453] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 8568.001061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8568.170393] Lustre: 240730:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8568.235165] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8568.241657] Lustre: Skipped 1 previous similar message [ 8568.327321] Lustre: lustre-MDT0001: new disk, initializing [ 8568.516793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8568.547797] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8573.339644] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 8584.447732] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8590.943637] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8598.432082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8598.758924] Lustre: lustre-OST0000: new disk, initializing [ 8598.768170] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8598.780506] Lustre: 242700:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8600.426282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8600.446388] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8600.589246] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8604.846272] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 8614.250791] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8618.963626] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8619.196431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8624.245834] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8624.496916] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8631.279594] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8631.279832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8631.284763] LustreError: Skipped 1 previous similar message [ 8631.286670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8631.307978] Lustre: Skipped 1 previous similar message [ 8632.547988] Lustre: server umount lustre-OST0000 complete [ 8641.506358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8641.519073] Lustre: Skipped 4 previous similar messages [ 8642.509587] Lustre: server umount lustre-MDT0000 complete [ 8645.961865] LustreError: 240723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272204 with bad export cookie 3175348121691666183 [ 8645.964598] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8645.972985] LustreError: 240723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8646.337794] Lustre: server umount lustre-MDT0001 complete [ 8655.739568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8656.529646] LustreError: 244915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8656.652731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8656.661139] Lustre: Skipped 2 previous similar messages [ 8661.703771] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 8667.133455] LustreError: 244915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8667.159703] LustreError: 244915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8673.368500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8678.618248] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 8687.005553] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8692.530390] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8699.510526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8707.229702] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 8715.192315] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8865.764197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8865.770901] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8865.780022] Lustre: Skipped 2 previous similar messages [ 8865.784668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8869.385503] Lustre: server umount lustre-OST0000 complete [ 8879.868517] Lustre: server umount lustre-MDT0000 complete [ 8883.168564] LustreError: 244915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8883.185467] LustreError: 244915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8883.504391] LustreError: 244898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272442 with bad export cookie 3175348121691667065 [ 8883.532032] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8883.539207] LustreError: 244898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8883.907988] Lustre: server umount lustre-MDT0001 complete [ 8984.992364] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [ 8988.566913] Key type lgssc unregistered [ 8989.101315] LNet: 248526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8989.110305] LNetError: 248526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8989.144763] LNet: Removed LNI 192.168.204.110@tcp [ 8990.492376] Key type .llcrypt unregistered [ 8990.495249] Key type ._llcrypt unregistered [ 9021.781589] Key type ._llcrypt registered [ 9021.785497] Key type .llcrypt registered [ 9021.984780] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid [ 9041.253579] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 9042.554346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9042.712563] alg: No test for adler32 (adler32-zlib) [ 9043.831543] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 9044.145240] LNet: Added LNI 192.168.204.110@tcp [8/256/0/180] [ 9045.903272] Key type lgssc registered [ 9047.089321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9101.795674] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 9112.568581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9112.598368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9113.797292] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9113.818710] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9113.875019] Lustre: lustre-MDT0000: new disk, initializing [ 9114.002315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9114.037154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9118.574622] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 9129.850637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9130.047785] Lustre: 252616:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9130.133162] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9130.145075] Lustre: Skipped 1 previous similar message [ 9130.333610] Lustre: lustre-MDT0001: new disk, initializing [ 9130.472043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9130.535724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9130.566712] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9136.479594] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 9141.773868] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9150.700691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9150.956886] Lustre: lustre-OST0000: new disk, initializing [ 9150.960677] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9150.965603] Lustre: 254249:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9151.043819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9151.497952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9151.509226] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9151.664700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9158.167365] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 9171.140856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9171.265516] Lustre: lustre-OST0001: new disk, initializing [ 9171.269912] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9171.275841] Lustre: 255122:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9171.356900] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9177.784123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9177.802199] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9177.974836] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9179.632533] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 9192.577564] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9201.331747] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9217.520063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9217.526048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9217.543835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9219.043734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9219.053187] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9219.069796] Lustre: Skipped 1 previous similar message [ 9223.191883] Lustre: server umount lustre-MDT0000 complete [ 9227.036140] LustreError: 255889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272785 with bad export cookie 18138711120225269039 [ 9227.064108] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9227.067069] LustreError: 255889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9227.638575] Lustre: server umount lustre-MDT0001 complete [ 9242.162052] Lustre: server umount lustre-OST0000 complete [ 9244.640339] Lustre: 250100:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789272787/real 1789272787] req@ffff8d52cce52d80 x1876188307997056/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789272803 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9244.679233] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9244.701864] Lustre: Skipped 1 previous similar message [ 9246.807561] Lustre: server umount lustre-OST0001 complete [ 9262.191647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9271.085628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9279.837469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9291.271542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9313.478911] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 9324.296252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9324.342722] Lustre: lustre-MDT0000: reset Object Index mappings [ 9324.817969] LustreError: 260132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9324.831972] LustreError: 260132:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9324.898790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9329.612544] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 9330.145815] LustreError: 260133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9335.273354] LustreError: 260132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9338.117523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9338.139124] Lustre: lustre-MDT0001: reset Object Index mappings [ 9338.591982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9342.746812] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 9345.579750] Lustre: 261274:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9351.729719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9359.241120] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 9364.328036] LustreError: 261628:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9367.955838] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9368.355689] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9368.364289] Lustre: Skipped 1 previous similar message [ 9370.452938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9370.485796] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9375.915097] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 9384.486545] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9388.618325] Lustre: 263147:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9394.656946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9394.664609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9394.680546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9394.684680] Lustre: Skipped 3 previous similar messages [ 9395.570482] Lustre: server umount lustre-MDT0000 complete [ 9396.194828] LustreError: 260133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9399.754389] LustreError: 260114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272958 with bad export cookie 18138711120225275829 [ 9399.760671] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9399.772916] LustreError: 260114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9400.311702] Lustre: server umount lustre-MDT0001 complete [ 9415.317545] Lustre: server umount lustre-OST0000 complete [ 9417.185901] Lustre: 250100:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789272959/real 1789272959] req@ffff8d52cbaf2680 x1876188308048256/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789272975 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9417.224245] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9417.235750] Lustre: Skipped 3 previous similar messages [ 9419.442435] Lustre: server umount lustre-OST0001 complete [ 9440.445865] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 00:16:37 (1789272997) [ 9458.439397] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid [ 9469.213708] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 9507.834183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9508.211562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9508.282177] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9508.363169] Lustre: lustre-MDT0000: new disk, initializing [ 9508.494541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9508.520290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9512.581064] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 9523.479389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9523.667223] Lustre: 267402:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9523.806898] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9523.819880] Lustre: Skipped 1 previous similar message [ 9523.898726] Lustre: lustre-MDT0001: new disk, initializing [ 9524.006849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9524.016740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9531.033475] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 9546.249781] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9554.763444] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9562.702638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9562.989596] Lustre: lustre-OST0000: new disk, initializing [ 9562.993943] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9563.001590] Lustre: 269372:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9564.841120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9564.856169] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9564.967457] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9570.093856] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 9582.987259] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9590.135191] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9590.432505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9596.041803] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9596.530795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9601.002572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9601.010809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9601.012070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9601.033383] Lustre: Skipped 1 previous similar message [ 9605.802010] Lustre: server umount lustre-OST0000 complete [ 9611.237964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9611.250892] Lustre: Skipped 1 previous similar message [ 9616.777464] Lustre: server umount lustre-MDT0000 complete [ 9621.137287] LustreError: 268196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789273179 with bad export cookie 18138711120225277922 [ 9621.151278] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9621.152199] LustreError: 268196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9621.791629] Lustre: server umount lustre-MDT0001 complete [ 9642.868834] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 9657.149755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9657.765474] LustreError: 272307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9657.831932] LustreError: 272307:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9657.881797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9657.889922] Lustre: Skipped 2 previous similar messages [ 9665.485523] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 9678.308958] LustreError: 272308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9678.332359] LustreError: 272308:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9680.293628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9686.168397] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 9689.938198] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9697.928702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9706.268876] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 9717.417626] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9717.662131] Lustre: lustre-OST0001: new disk, initializing [ 9717.665878] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9717.676483] Lustre: 274620:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9719.123465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9719.139260] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9719.236096] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9724.455748] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 9736.068605] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9740.830777] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9747.793234] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9747.862059] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9757.676677] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9757.687832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9757.697530] Lustre: Skipped 3 previous similar messages [ 9757.716098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9757.729192] Lustre: Skipped 2 previous similar messages [ 9762.463764] Lustre: server umount lustre-MDT0000 complete [ 9765.350102] LustreError: 272307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9765.382120] LustreError: 272307:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9767.407101] LustreError: 273043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789273325 with bad export cookie 18138711120225278804 [ 9767.416152] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9767.423878] LustreError: 273043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9768.164266] Lustre: server umount lustre-MDT0001 complete [ 9784.585942] Lustre: server umount lustre-OST0000 complete [ 9786.863823] Lustre: 250102:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789273329/real 1789273329] req@ffff8d53ef41fb80 x1876188308162304/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789273345 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9788.903302] Lustre: 250100:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789273331/real 1789273331] req@ffff8d52c905f480 x1876188308162560/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789273347 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9789.250112] Lustre: server umount lustre-OST0001 complete [ 9810.956765] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [ 9833.633088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9834.613065] LustreError: 277678:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9834.794527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9834.796761] Lustre: Skipped 3 previous similar messages [ 9841.375168] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 9852.607637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9859.762352] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 9863.996683] Lustre: 278820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9871.981315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9879.756965] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 9879.795911] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9890.157653] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9895.972856] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9900.168280] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [ 9911.595562] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9917.436936] Lustre: 280696:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9924.378605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9924.431599] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9937.647221] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 00:24:53 (1789273493) [ 9955.305360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9955.324312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9955.356061] Lustre: Skipped 4 previous similar messages [ 9955.362364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9955.375505] Lustre: Skipped 4 previous similar messages [ 9960.674474] Lustre: server umount lustre-MDT0000 complete [ 9966.295104] LustreError: 277658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789273524 with bad export cookie 18138711120225285118 [ 9966.312060] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9967.589669] LustreError: 277675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9967.613580] LustreError: 277675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 9972.841302] Lustre: server umount lustre-MDT0001 complete [ 9984.651283] Lustre: server umount lustre-OST0000 complete [ 9989.545288] Lustre: server umount lustre-OST0001 complete [ 9999.956396] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid [10017.557278] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [10077.644574] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [10090.305989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10090.708034] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10090.742585] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10090.831384] Lustre: lustre-MDT0000: new disk, initializing [10090.885965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10090.893586] Lustre: Skipped 3 previous similar messages [10090.917556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10098.834577] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [10114.372791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10114.741392] Lustre: 284962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10115.345830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10121.525775] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [10127.972736] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10138.432433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10138.697838] Lustre: 286596:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10140.824974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10148.312874] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [10165.315107] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10165.597633] Lustre: 287470:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10167.266508] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10167.286364] Lustre: Skipped 1 previous similar message [10167.442547] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10174.635741] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [10186.707950] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10212.625019] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 00:29:28 (1789273768) [10217.440902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10217.449499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10217.477577] Lustre: Skipped 5 previous similar messages [10217.484658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10217.495702] Lustre: Skipped 8 previous similar messages [10230.239849] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10230.670221] Lustre: server umount lustre-MDT0000 complete [10233.832954] LustreError: 284976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10233.859990] LustreError: 284976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10235.585996] LustreError: 284955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789273794 with bad export cookie 18138711120225292993 [10235.602588] LustreError: 284955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10235.607509] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10235.911885] Lustre: server umount lustre-MDT0001 complete [10251.600309] Lustre: server umount lustre-OST0000 complete [10255.199149] Lustre: 250102:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789273797/real 1789273797] req@ffff8d53f69ac000 x1876188308334976/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789273813 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10258.330533] Lustre: server umount lustre-OST0001 complete [10272.608062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10279.730605] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [10291.292614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10298.774698] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [10308.714352] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10315.320800] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10324.368473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10332.748550] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [10341.711877] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10361.938292] loop1: detected capacity change from 0 to 204800000 [10362.149269] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10362.167777] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10362.191948] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10368.168857] loop1: detected capacity change from 0 to 204800000 [10375.932331] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10376.047449] LustreError: 289709:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs5908: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10376.062560] LustreError: 289709:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs5908 [10376.067230] LustreError: 289709:0:(mgs_handler.c:517:mgs_target_reg()) fs5908-MDT0001: Error checking target index: rc = -13 [10376.098816] LustreError: fs5908-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10376.110840] LustreError: 293445:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [10376.126855] LustreError: 293445:0:(tgt_mount.c:2132:server_put_super()) no obd fs5908-MDT0001 [10376.133458] LustreError: 293445:0:(tgt_mount.c:116:server_deregister_mount()) fs5908-MDT0001 not registered [10376.192150] Lustre: server umount fs5908-MDT0001 complete [10376.194946] LustreError: 293445:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10387.769193] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10387.897740] Lustre: Setting parameter fs5908-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs5908-MDT0001 [10387.906721] Lustre: Skipped 1 previous similar message [10387.944458] Lustre: srv-fs5908-MDT0001: No data found on store. Initialize space. [10387.952177] Lustre: Skipped 4 previous similar messages [10388.084098] Lustre: fs5908-MDT0001: new disk, initializing [10388.086087] Lustre: Skipped 3 previous similar messages [10388.190610] Lustre: srv-fs5908-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10388.207769] Lustre: 293912:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5908-MDT0001-mdtlov: can't create new llog: rc = -115 [10388.226932] LustreError: 293912:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5908-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10393.433681] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [10403.963643] Lustre: server umount fs5908-MDT0001 complete [10415.082134] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10419.190822] loop1: detected capacity change from 0 to 204800000 [10425.597434] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10426.089798] Lustre: 295481:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5908-MDT0001-mdtlov: can't create new llog: rc = -115 [10426.107555] LustreError: 295481:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5908-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10430.719513] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [10435.655300] Lustre: server umount fs5908-MDT0001 complete [10449.984436] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.10@tcp (stopping) [10449.992507] Lustre: Skipped 12 previous similar messages [10450.913403] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10450.943114] Lustre: Skipped 4 previous similar messages [10450.943244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10450.964278] LustreError: Skipped 1 previous similar message [10455.472519] Lustre: server umount lustre-MDT0000 complete [10466.645552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10466.856460] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10472.036626] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [10472.434640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10480.811861] loop1: detected capacity change from 0 to 204800000 [10487.030191] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10487.587474] Lustre: 297925:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5908-MDT0001-mdtlov: can't create new llog: rc = -115 [10487.600288] LustreError: 297925:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5908-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10493.037259] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [10498.130382] Lustre: server umount fs5908-MDT0001 complete [10523.662654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10528.185916] Lustre: server umount lustre-OST0000 complete [10538.977838] Lustre: server umount lustre-MDT0000 complete [10544.607232] LustreError: 289701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789274103 with bad export cookie 18138711120225295674 [10544.617034] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10544.620459] LustreError: 289701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10545.322142] Lustre: server umount lustre-MDT0001 complete [10565.671252] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [10568.345770] Key type lgssc unregistered [10568.748552] LNet: 300291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10568.761209] LNetError: 300291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10569.830761] LNet: Removed LNI 192.168.204.110@tcp [10570.831851] Key type .llcrypt unregistered [10570.836479] Key type ._llcrypt unregistered [10594.881706] Key type ._llcrypt registered [10594.884399] Key type .llcrypt registered [10595.033596] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [10596.052041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10596.170039] alg: No test for adler32 (adler32-zlib) [10597.421806] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [10597.703059] LNet: Added LNI 192.168.204.110@tcp [8/256/0/180] [10599.385765] Key type lgssc registered [10600.695299] Lustre: Echo OBD driver; http://www.lustre.org/ [10608.443414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10608.471418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10610.052350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10616.394847] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [10620.410172] LustreError: 301561:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10620.438234] LustreError: 301561:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10625.508415] LustreError: 301560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10627.289405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10627.887870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10632.436092] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [10642.047436] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10649.058940] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10657.362097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10657.701634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10663.695359] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [10672.301814] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10673.072404] LustreError: 303393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10678.244344] LustreError: 303394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10683.479431] LustreError: 304064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10683.509876] LustreError: 304064:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10693.646980] LustreError: 304064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10693.691410] LustreError: 304064:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10702.795799] loop1: detected capacity change from 0 to 204800000 [10702.974726] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10703.012960] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10703.043813] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10708.889578] loop1: detected capacity change from 0 to 204800000 [10711.013024] LustreError: 304064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10711.030079] LustreError: 304064:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [10718.287463] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10718.650495] Lustre: fs21384-OST0000: new disk, initializing [10718.667389] Lustre: srv-fs21384-OST0000: No data found on store. Initialize space. [10718.687968] Lustre: 305320:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10718.874234] Lustre: fs21384-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10728.951374] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [10740.675751] Lustre: server umount fs21384-OST0000 complete [10744.829475] LustreError: 303393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10744.865740] LustreError: 303393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [10751.979057] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10752.652303] loop1: detected capacity change from 0 to 204800000 [10757.110425] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10758.834549] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10759.185343] Lustre: fs21384-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10766.231807] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [10775.835564] loop2: detected capacity change from 0 to 204800000 [10776.157172] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10776.179549] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10776.193682] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10782.917790] loop2: detected capacity change from 0 to 204800000 [10790.269482] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10790.404693] LustreError: fs21384-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10790.420900] LustreError: fs21384-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10790.427984] LustreError: 308365:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [10790.433255] LustreError: 308365:0:(tgt_mount.c:2132:server_put_super()) no obd fs21384-OST0001 [10790.437114] LustreError: 308365:0:(tgt_mount.c:116:server_deregister_mount()) fs21384-OST0001 not registered [10790.488330] Lustre: server umount fs21384-OST0001 complete [10790.493109] LustreError: 308365:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10803.579981] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10803.732796] Lustre: fs21384-OST0001: new disk, initializing [10803.736503] Lustre: srv-fs21384-OST0001: No data found on store. Initialize space. [10803.742493] Lustre: 308917:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10803.830231] Lustre: fs21384-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10810.410967] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [10811.404085] LustreError: 306947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10811.437886] LustreError: 306947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [10820.797889] Lustre: server umount fs21384-OST0000 complete [10831.471215] Lustre: server umount fs21384-OST0001 complete [10845.664517] loop1: detected capacity change from 0 to 204800000 [10852.160518] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10852.491216] Lustre: fs21384-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10859.853654] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [10867.051131] loop2: detected capacity change from 0 to 204800000 [10872.948844] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10880.568718] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [10890.909872] Lustre: server umount fs21384-OST0000 complete [10901.208186] Lustre: server umount fs21384-OST0001 complete [10905.603739] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10910.708764] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10915.830573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10915.837770] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10915.866325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10915.886225] Lustre: Skipped 1 previous similar message [10920.860385] Lustre: server umount lustre-OST0000 complete [10926.051790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10926.070699] Lustre: Skipped 1 previous similar message [10926.074917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10926.091613] Lustre: Skipped 1 previous similar message [10931.169704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10931.185208] Lustre: Skipped 1 previous similar message [10936.291314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10940.383486] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10941.410476] LustreError: 301557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10941.459580] LustreError: 301557:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 62 previous similar messages [10941.541201] Lustre: server umount lustre-MDT0000 complete [10947.014473] LustreError: 301541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789274505 with bad export cookie 105304840617574404 [10947.017911] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10947.030871] LustreError: 301541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10947.331375] Lustre: server umount lustre-MDT0001 complete [10961.448607] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [10965.217795] Key type lgssc unregistered [10965.578680] LNet: 314844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10965.584515] LNetError: 314844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10965.605164] LNet: Removed LNI 192.168.204.110@tcp [10966.564158] Key type .llcrypt unregistered [10966.565796] Key type ._llcrypt unregistered [11006.478863] Key type ._llcrypt registered [11006.485117] Key type .llcrypt registered [11006.753682] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid [11023.238888] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [11024.816588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11024.904546] alg: No test for adler32 (adler32-zlib) [11026.138802] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [11026.336944] LNet: Added LNI 192.168.204.110@tcp [8/256/0/180] [11028.007218] Key type lgssc registered [11029.396304] Lustre: Echo OBD driver; http://www.lustre.org/ [11070.372268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11070.397055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11071.775862] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11071.814920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11071.875585] Lustre: lustre-MDT0000: new disk, initializing [11072.031952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11072.048552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11077.118241] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [11090.358359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11090.425953] Lustre: 318702:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11090.474788] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11090.478229] Lustre: Skipped 1 previous similar message [11090.547766] Lustre: lustre-MDT0001: new disk, initializing [11090.625331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11090.731823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11090.749928] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11097.282057] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [11109.092368] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11115.414657] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11122.958458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11123.461152] Lustre: lustre-OST0000: new disk, initializing [11123.466232] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11123.478414] Lustre: 320672:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11123.644118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11131.264170] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [11132.496219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11132.503993] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11132.625396] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11142.259333] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11144.957449] LustreError: 316874:0:(lmv_obd.c:211:lmv_notify()) activation of lustre-MDT0001_UUID failed: -22 [11145.002163] Lustre: Mounted lustre-client - version 2.17.57_103_g3c4c45f [11146.267873] Lustre: Unmounted lustre-client [11158.510738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11158.511234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11158.517769] LustreError: Skipped 1 previous similar message [11158.522755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11158.529484] Lustre: Skipped 1 previous similar message [11158.545265] Lustre: Skipped 1 previous similar message [11160.555834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11160.562809] Lustre: Skipped 1 previous similar message [11164.531685] Lustre: server umount lustre-OST0000 complete [11170.788399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11170.814622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11170.823789] Lustre: Skipped 1 previous similar message [11175.925654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11175.937488] Lustre: Skipped 1 previous similar message [11176.048346] Lustre: server umount lustre-MDT0000 complete [11180.148187] LustreError: 319497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789274738 with bad export cookie 2777144027509554774 [11180.149098] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11180.161185] LustreError: 319497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11180.490350] Lustre: server umount lustre-MDT0001 complete [11193.966418] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [11196.322529] LNet: 1 local NIs in recovery (showing 1): 192.168.204.110@tcp [11197.333923] Key type lgssc unregistered [11197.727233] LNet: 322833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11197.744520] LNetError: 322833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11273.605214] LNet: Removed LNI 192.168.204.110@tcp [11274.459184] Key type .llcrypt unregistered [11274.464736] Key type ._llcrypt unregistered [11294.614270] Key type ._llcrypt registered [11294.620870] Key type .llcrypt registered [11294.881783] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid [11313.836797] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [11314.444715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11314.460084] alg: No test for adler32 (adler32-zlib) [11315.546427] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [11315.797026] LNet: Added LNI 192.168.204.110@tcp [8/256/0/180] [11317.519166] Key type lgssc registered [11318.914045] Lustre: Echo OBD driver; http://www.lustre.org/ [11355.362879] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 00:48:31 (1789274911) [11362.231622] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11362.245715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11363.506845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11363.534305] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11363.592228] Lustre: lustre-MDT0000: new disk, initializing [11363.664246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11363.692511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11367.927497] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [11378.861672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11378.995806] Lustre: 326055:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11379.076381] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11379.080867] Lustre: Skipped 1 previous similar message [11379.234293] Lustre: lustre-MDT0001: new disk, initializing [11379.320307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11379.361200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11379.373217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11384.993434] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [11395.434206] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11402.372558] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11410.185305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11410.475940] Lustre: lustre-OST0000: new disk, initializing [11410.480474] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11410.488372] Lustre: 328026:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11410.579677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11417.343060] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [11419.211565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11419.233041] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11419.347075] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11426.808996] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11445.216971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11445.218218] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11445.231682] LustreError: Skipped 1 previous similar message [11445.232875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11445.272953] Lustre: Skipped 1 previous similar message [11450.343692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11450.351937] Lustre: Skipped 2 previous similar messages [11451.284318] Lustre: server umount lustre-OST0000 complete [11455.459948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11455.478016] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11460.580452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11460.587521] Lustre: Skipped 2 previous similar messages [11461.700921] Lustre: server umount lustre-MDT0000 complete [11465.622507] LustreError: 326048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275024 with bad export cookie 11283717935500132986 [11465.623155] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11465.626415] LustreError: 326048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11465.865150] Lustre: server umount lustre-MDT0001 complete [11478.544552] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [11481.791714] Key type lgssc unregistered [11482.044699] LNet: 329935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11482.051895] LNetError: 329935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11483.110375] LNet: Removed LNI 192.168.204.110@tcp [11484.222598] Key type .llcrypt unregistered [11484.226228] Key type ._llcrypt unregistered [11511.369033] Key type ._llcrypt registered [11511.374113] Key type .llcrypt registered [11511.568786] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [11513.173962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11513.267639] alg: No test for adler32 (adler32-zlib) [11514.431439] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [11514.716238] LNet: Added LNI 192.168.204.110@tcp [8/256/0/180] [11516.679313] Key type lgssc registered [11518.598911] Lustre: Echo OBD driver; http://www.lustre.org/ [11530.471475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11530.484807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11532.110421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11538.097125] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [11542.512839] LustreError: 331209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11542.531204] LustreError: 331209:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11547.617159] LustreError: 331210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11548.217120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11549.058299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11554.242133] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [11563.678307] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11570.593818] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11580.224966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11580.926958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11587.950907] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [11597.039289] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11612.648619] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11614.156897] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11618.786505] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11618.791411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11618.796426] Lustre: Skipped 2 previous similar messages [11618.808303] Lustre: Skipped 2 previous similar messages [11619.375827] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.10@tcp (stopping) [11623.911954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11623.921819] Lustre: Skipped 2 previous similar messages [11629.030137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11629.035370] Lustre: Skipped 3 previous similar messages [11632.095240] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11632.413381] Lustre: server umount lustre-MDT0000 complete [11634.151415] LustreError: 331204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11634.191230] LustreError: 331204:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11637.948253] LustreError: 331189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275196 with bad export cookie 13268231909388919031 [11637.971510] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11639.269209] LustreError: 332372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11639.271119] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11639.302333] LustreError: 332372:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11639.328490] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11639.332884] Lustre: Skipped 1 previous similar message [11644.882414] Lustre: server umount lustre-MDT0001 complete [11649.699766] Lustre: server umount lustre-OST0000 complete [11659.276030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11664.937400] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [11668.096138] Lustre: 335396:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.110@tcp [11670.430686] Lustre: server umount MGS complete [11679.343755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11679.763243] LustreError: 335962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11679.797074] LustreError: 335962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11679.908044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11684.623923] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [11689.992920] LustreError: 335962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11690.026990] LustreError: 335962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11692.576320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11692.890186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11697.694078] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [11708.447667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11708.956552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11717.096137] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [11726.058584] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11727.893830] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11734.440712] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11736.362322] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11742.189462] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11742.202037] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11742.227384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11742.231589] Lustre: Skipped 2 previous similar messages [11743.744698] Lustre: server umount lustre-OST0000 complete [11749.345081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11749.355919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11754.585732] Lustre: server umount lustre-MDT0000 complete [11756.001457] LustreError: 336695:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11756.028248] LustreError: 336695:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11759.415593] LustreError: 335948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275317 with bad export cookie 13268231909388919871 [11759.419890] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11759.425095] LustreError: 335948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11768.927227] Lustre: server umount lustre-MDT0001 complete [11786.599222] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [11791.186480] Key type lgssc unregistered [11791.602242] LNet: 339405:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11791.614446] LNetError: 339405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11791.674456] LNet: Removed LNI 192.168.204.110@tcp [11793.392789] Key type .llcrypt unregistered [11793.396491] Key type ._llcrypt unregistered [11815.840715] Key type ._llcrypt registered [11815.845253] Key type .llcrypt registered [11815.981588] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid [11833.024460] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [11833.985074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11834.183405] alg: No test for adler32 (adler32-zlib) [11835.618985] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [11835.946678] LNet: Added LNI 192.168.204.110@tcp [8/256/0/180] [11837.655276] Key type lgssc registered [11838.845042] Lustre: Echo OBD driver; http://www.lustre.org/ [11873.624354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11873.637832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11874.895614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11874.916552] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11874.981937] Lustre: lustre-MDT0000: new disk, initializing [11875.091341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11875.120414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11881.915582] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [11897.126719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11897.199893] Lustre: 342568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11897.245120] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11897.249344] Lustre: Skipped 1 previous similar message [11897.294187] Lustre: lustre-MDT0001: new disk, initializing [11897.351168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11897.377763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11897.387369] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11902.593966] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [11914.188534] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11921.144926] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11928.237620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11928.610315] Lustre: lustre-OST0000: new disk, initializing [11928.613910] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11928.620426] Lustre: 344537:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11928.692797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11934.161375] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [11936.386655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11936.410713] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11936.556525] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11944.521691] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11949.252509] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11949.480687] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11955.776081] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11956.209979] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11962.342688] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11962.348936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11962.370997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11962.376480] Lustre: Skipped 1 previous similar message [11964.433710] Lustre: server umount lustre-OST0000 complete [11968.994488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11969.013243] LustreError: Skipped 1 previous similar message [11969.025121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11969.052889] Lustre: Skipped 1 previous similar message [11969.068855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11972.578135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11972.579525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11977.701116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11977.715855] Lustre: Skipped 2 previous similar messages [11981.794346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11981.802027] Lustre: Skipped 1 previous similar message [11983.327293] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11983.545626] Lustre: server umount lustre-MDT0000 complete [11987.501341] LustreError: 342559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275546 with bad export cookie 13389090017066517192 [11987.509247] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11987.517807] LustreError: 342559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11987.965672] Lustre: server umount lustre-MDT0001 complete [11997.650530] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 00:59:14 (1789275554) [12012.547813] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid [12022.937834] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [12066.269660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12066.607746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12066.662460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12066.755773] Lustre: lustre-MDT0000: new disk, initializing [12066.886390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12066.909878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12071.577721] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [12083.198803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12083.338715] Lustre: 349297:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12083.427492] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12083.431543] Lustre: Skipped 1 previous similar message [12083.479539] Lustre: lustre-MDT0001: new disk, initializing [12083.563322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12083.609187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12083.620196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12088.398536] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [12099.507446] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12106.065799] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12113.094957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12113.323935] Lustre: lustre-OST0000: new disk, initializing [12113.327178] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12113.334566] Lustre: 351278:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12113.429622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12120.442472] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [12122.298591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12122.308606] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12122.480675] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12135.532823] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12173.806650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12173.809521] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12173.812612] LustreError: Skipped 1 previous similar message [12173.814109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12173.828886] Lustre: Skipped 1 previous similar message [12175.492920] Lustre: server umount lustre-OST0000 complete [12180.961635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12180.976047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12186.976256] Lustre: server umount lustre-MDT0000 complete [12189.153049] LustreError: 349303:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12189.169836] LustreError: 349303:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12190.510204] LustreError: 350101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275749 with bad export cookie 13389090017066518074 [12190.518736] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12190.843350] Lustre: server umount lustre-MDT0001 complete [12206.016690] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [12209.307904] Key type lgssc unregistered [12209.897963] LNet: 353434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12209.916948] LNetError: 353434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12209.934184] LNet: Removed LNI 192.168.204.110@tcp [12211.205135] Key type .llcrypt unregistered [12211.207342] Key type ._llcrypt unregistered [12314.335762] Key type ._llcrypt registered [12314.343181] Key type .llcrypt registered [12314.529275] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [12314.930512] Key type .llcrypt unregistered [12314.933613] Key type ._llcrypt unregistered [12408.811886] Key type ._llcrypt registered [12408.815428] Key type .llcrypt registered [12408.966533] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [12409.379418] Key type .llcrypt unregistered [12409.384708] Key type ._llcrypt unregistered [12442.575155] Key type ._llcrypt registered [12442.579052] Key type .llcrypt registered [12442.693723] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [12442.960190] Key type .llcrypt unregistered [12442.963272] Key type ._llcrypt unregistered [12476.846761] Key type ._llcrypt registered [12476.850229] Key type .llcrypt registered [12476.966775] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [12477.367440] Key type .llcrypt unregistered [12477.372674] Key type ._llcrypt unregistered [12564.854375] Key type ._llcrypt registered [12564.856240] Key type .llcrypt registered [12564.938980] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [12565.317153] Key type .llcrypt unregistered [12565.318874] Key type ._llcrypt unregistered [12602.087884] Key type ._llcrypt registered [12602.100858] Key type .llcrypt registered [12602.303714] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [12602.611868] Key type .llcrypt unregistered [12602.618821] Key type ._llcrypt unregistered [12639.020174] Key type ._llcrypt registered [12639.022234] Key type .llcrypt registered [12639.146286] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [12639.541630] Key type .llcrypt unregistered [12639.544419] Key type ._llcrypt unregistered [12671.439920] Key type ._llcrypt registered [12671.441967] Key type .llcrypt registered [12671.573451] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [12671.970117] Key type .llcrypt unregistered [12671.973457] Key type ._llcrypt unregistered [12706.558757] Key type ._llcrypt registered [12706.564173] Key type .llcrypt registered [12706.665696] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [12707.026934] Key type .llcrypt unregistered [12707.038177] Key type ._llcrypt unregistered [12741.682476] Key type ._llcrypt registered [12741.685420] Key type .llcrypt registered [12741.868164] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [12742.264389] Key type .llcrypt unregistered [12742.273465] Key type ._llcrypt unregistered [12822.212629] Key type ._llcrypt registered [12822.216861] Key type .llcrypt registered [12822.346628] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [12822.827319] Key type .llcrypt unregistered [12822.829036] Key type ._llcrypt unregistered [12858.699467] Key type ._llcrypt registered [12858.702467] Key type .llcrypt registered [12858.877597] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [12859.204536] Key type .llcrypt unregistered [12859.211650] Key type ._llcrypt unregistered [12894.032886] Key type ._llcrypt registered [12894.042677] Key type .llcrypt registered [12894.223893] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [12894.698638] Key type .llcrypt unregistered [12894.704753] Key type ._llcrypt unregistered [12959.676243] Key type ._llcrypt registered [12959.679355] Key type .llcrypt registered [12959.811713] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid [12977.522550] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [12979.139218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12979.418378] alg: No test for adler32 (adler32-zlib) [12980.745303] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [12981.088533] LNet: Added LNI 192.168.204.110@tcp [8/256/0/180] [12982.831261] Key type lgssc registered [12984.123201] Lustre: Echo OBD driver; http://www.lustre.org/ [13027.790917] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13027.823883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13029.168478] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13029.208497] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13029.312919] Lustre: tst250fs-MDT0000: new disk, initializing [13029.402375] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13029.419923] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13035.102310] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [13047.830379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13048.022134] Lustre: 373732:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13048.080507] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13048.084684] Lustre: Skipped 1 previous similar message [13048.174478] Lustre: tst250fs-MDT0001: new disk, initializing [13048.346718] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13048.417043] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13048.438452] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13053.751582] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [13058.951403] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13066.395301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13066.644248] Lustre: tst250fs-OST0000: new disk, initializing [13066.648314] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13066.654085] Lustre: 375362:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13066.742798] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13067.117177] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13067.130776] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13067.321302] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13073.167744] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [13084.473571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13084.652389] Lustre: tst250fs-OST0001: new disk, initializing [13084.656183] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13084.662766] Lustre: 376235:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13084.775596] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13091.234643] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug -1 all [13092.018433] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13092.032775] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13092.142706] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13097.642200] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13099.894194] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13099.906678] Lustre: Skipped 1 previous similar message [13119.968572] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13119.974312] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13119.992591] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13123.041520] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13123.052466] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13128.171324] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13128.187540] Lustre: Skipped 3 previous similar messages [13133.289667] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13133.319110] Lustre: Skipped 6 previous similar messages [13134.303143] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13134.690334] Lustre: server umount tst250fs-MDT0000 complete [13139.058074] LustreError: 373724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789276697 with bad export cookie 5304291415239678150 [13139.061182] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13139.074718] LustreError: 373724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13139.630882] Lustre: server umount tst250fs-MDT0001 complete [13154.935199] Lustre: server umount tst250fs-OST0000 complete [13168.989700] Lustre: server umount tst250fs-OST0001 complete [13191.679061] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid [13202.411519] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [13238.280474] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 01:19:55 (1789276795) [13240.194938] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13262.683396] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid [13275.366400] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local [13314.447259] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13181 sec ========= 01:21:10 (1789276870) [13316.356418] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:21:12 (1789276872) === [13320.297582] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:21:16 (1789276876) === [13360.231834] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing unload_modules_local [13363.517771] Key type lgssc unregistered [13363.901659] LNet: 385381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13363.918771] LNetError: 385381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13363.954733] LNet: Removed LNI 192.168.204.110@tcp [13364.764197] Key type .llcrypt unregistered [13364.766393] Key type ._llcrypt unregistered