[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 518333122 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003325] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.007782] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010129] LSM: Security Framework initializing [ 0.011055] Yama: becoming mindful. [ 0.012033] SELinux: Initializing. [ 0.013063] *** VALIDATE selinux *** [ 0.021410] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025744] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026155] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028107] *** VALIDATE tmpfs *** [ 0.029448] *** VALIDATE proc *** [ 0.030263] *** VALIDATE cgroup *** [ 0.031010] *** VALIDATE cgroup2 *** [ 0.032290] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035026] Spectre V2 : User space: Vulnerable [ 0.036008] Speculative Store Bypass: Vulnerable [ 0.039391] debug: unmapping init [mem 0xffffffff8e659000-0xffffffff8e660fff] [ 0.041172] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042765] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043026] ... version: 2 [ 0.044015] ... bit width: 48 [ 0.045012] ... generic registers: 4 [ 0.046012] ... value mask: 0000ffffffffffff [ 0.047016] ... max period: 00007fffffffffff [ 0.048018] ... fixed-purpose events: 3 [ 0.049013] ... event mask: 000000070000000f [ 0.050296] rcu: Hierarchical SRCU implementation. [ 0.052364] smp: Bringing up secondary CPUs ... [ 0.053573] x86: Booting SMP configuration: [ 0.054026] .... node #0, CPUs: #1 #2 #3 [ 0.058034] smp: Brought up 1 node, 4 CPUs [ 0.060023] smpboot: Max logical packages: 1 [ 0.061013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.164089] node 0 deferred pages initialised in 99ms [ 0.169147] devtmpfs: initialized [ 0.170270] x86/mm: Memory block size: 128MB [ 0.174025] gcov: version magic: 0x41383552 [ 0.176228] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.180101] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.182282] pinctrl core: initialized pinctrl subsystem [ 0.184192] [ 0.184734] ************************************************************* [ 0.187015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.190013] ** ** [ 0.192013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.194013] ** ** [ 0.196014] ** This means that this kernel is built to expose internal ** [ 0.199012] ** IOMMU data structures, which may compromise security on ** [ 0.201016] ** your system. ** [ 0.204014] ** ** [ 0.206013] ** If you see this message and you are not debugging the ** [ 0.209014] ** kernel, report this immediately to your vendor! ** [ 0.211011] ** ** [ 0.214015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.216010] ************************************************************* [ 0.218690] NET: Registered protocol family 16 [ 0.221408] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.223058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.226061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.230040] cpuidle: using governor menu [ 0.231739] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.234445] PCI: Using configuration type 1 for base access [ 0.237229] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.246158] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.249038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.254099] cryptd: max_cpu_qlen set to 1000 [ 0.256273] ACPI: Added _OSI(Module Device) [ 0.257013] ACPI: Added _OSI(Processor Device) [ 0.258031] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.260029] ACPI: Added _OSI(Processor Aggregator Device) [ 0.268000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.278160] ACPI: Interpreter enabled [ 0.280076] ACPI: PM: (supports S0 S3 S4 S5) [ 0.281024] ACPI: Using IOAPIC for interrupt routing [ 0.283180] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.287475] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.299119] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.302062] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.306038] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.309105] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.313964] acpiphp: Slot [2] registered [ 0.315125] acpiphp: Slot [5] registered [ 0.316165] acpiphp: Slot [6] registered [ 0.318165] acpiphp: Slot [7] registered [ 0.319101] acpiphp: Slot [8] registered [ 0.321116] acpiphp: Slot [9] registered [ 0.323134] acpiphp: Slot [10] registered [ 0.325152] acpiphp: Slot [3] registered [ 0.326129] acpiphp: Slot [4] registered [ 0.329124] acpiphp: Slot [11] registered [ 0.330767] acpiphp: Slot [12] registered [ 0.333096] acpiphp: Slot [13] registered [ 0.334112] acpiphp: Slot [14] registered [ 0.336160] acpiphp: Slot [15] registered [ 0.337138] acpiphp: Slot [16] registered [ 0.339127] acpiphp: Slot [17] registered [ 0.340133] acpiphp: Slot [18] registered [ 0.342216] acpiphp: Slot [19] registered [ 0.343130] acpiphp: Slot [20] registered [ 0.345188] acpiphp: Slot [21] registered [ 0.347133] acpiphp: Slot [22] registered [ 0.348408] acpiphp: Slot [23] registered [ 0.350144] acpiphp: Slot [24] registered [ 0.352250] acpiphp: Slot [25] registered [ 0.353146] acpiphp: Slot [26] registered [ 0.355146] acpiphp: Slot [27] registered [ 0.357151] acpiphp: Slot [28] registered [ 0.359145] acpiphp: Slot [29] registered [ 0.360139] acpiphp: Slot [30] registered [ 0.362132] acpiphp: Slot [31] registered [ 0.364089] PCI host bridge to bus 0000:00 [ 0.366037] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.369054] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.371028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.374025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.377023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.380026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.383184] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.386105] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.390330] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.402016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.407987] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.411021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.414018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.417020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.420601] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.423819] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.427063] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.429805] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.435023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.449033] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.456020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.461759] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.471017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.481019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.503017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.514043] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.524022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.531021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.553028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.563732] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.572018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.581016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.598026] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.610102] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.619022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.628022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.652033] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.664948] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.676017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.684017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.712016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.727883] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.736022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.745016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.766020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.778053] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.779368] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.783394] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.786494] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.789253] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.796042] iommu: Default domain type: Passthrough [ 0.799610] SCSI subsystem initialized [ 0.801180] ACPI: bus type USB registered [ 0.803174] usbcore: registered new interface driver usbfs [ 0.805183] usbcore: registered new interface driver hub [ 0.808127] usbcore: registered new device driver usb [ 0.810231] pps_core: LinuxPPS API ver. 1 registered [ 0.812023] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.816093] PTP clock support registered [ 0.818377] EDAC MC: Ver: 3.0.0 [ 0.821136] PCI: Using ACPI for IRQ routing [ 0.823098] NetLabel: Initializing [ 0.824000] NetLabel: domain hash size = 128 [ 0.827022] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.829107] NetLabel: unlabeled traffic allowed by default [ 0.832166] vgaarb: loaded [ 0.833323] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.835026] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.842000] clocksource: Switched to clocksource kvm-clock [ 0.953717] VFS: Disk quotas dquot_6.6.0 [ 0.955699] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.958415] *** VALIDATE ramfs *** [ 0.959642] *** VALIDATE hugetlbfs *** [ 0.961564] pnp: PnP ACPI init [ 0.964778] pnp: PnP ACPI: found 6 devices [ 0.980742] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.985197] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.987592] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.989798] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.992204] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.994707] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.997569] NET: Registered protocol family 2 [ 1.000709] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.005540] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.009102] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.014727] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.018598] TCP: Hash tables configured (established 65536 bind 65536) [ 1.021856] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.025147] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.028438] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.031147] NET: Registered protocol family 1 [ 1.033298] RPC: Registered named UNIX socket transport module. [ 1.035785] RPC: Registered udp transport module. [ 1.037818] RPC: Registered tcp transport module. [ 1.039744] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.042367] NET: Registered protocol family 44 [ 1.044205] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.046177] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.048122] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.052554] PCI: CLS 0 bytes, default 64 [ 1.054486] Unpacking initramfs... [ 2.541431] debug: unmapping init [mem 0xffff9766bcc54000-0xffff9766bffbffff] [ 2.546809] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.550290] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.554670] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.089256] Initialise system trusted keyrings [ 3.091408] Key type blacklist registered [ 3.093604] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.103474] zbud: loaded [ 3.106615] *** VALIDATE nfs *** [ 3.107830] *** VALIDATE nfs4 *** [ 3.109697] pstore: using deflate compression [ 3.112223] Platform Keyring initialized [ 3.228750] NET: Registered protocol family 38 [ 3.230552] Key type asymmetric registered [ 3.231993] Asymmetric key parser 'x509' registered [ 3.233988] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.237579] io scheduler mq-deadline registered [ 3.240083] io scheduler kyber registered [ 3.241589] io scheduler bfq registered [ 3.243823] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.246160] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.248962] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.252219] ACPI: Power Button [PWRF] [ 3.258052] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.265574] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.296232] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.308872] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.345810] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.374367] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.402446] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.407939] Non-volatile memory driver v1.3 [ 3.409894] Linux agpgart interface v0.103 [ 3.442859] virtio_blk virtio1: [vda] 149456 512-byte logical blocks (76.5 MB/73.0 MiB) [ 3.446269] vda: detected capacity change from 0 to 76521472 [ 3.462641] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.465617] vdb: detected capacity change from 0 to 1073741824 [ 3.481390] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.484676] vdc: detected capacity change from 0 to 2621440000 [ 3.500537] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.504283] vdd: detected capacity change from 0 to 2621440000 [ 3.519276] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.522684] vde: detected capacity change from 0 to 4294967296 [ 3.538138] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.541590] vdf: detected capacity change from 0 to 4294967296 [ 3.549081] libphy: Fixed MDIO Bus: probed [ 3.553590] usbcore: registered new interface driver usbserial_generic [ 3.555595] usbserial: USB Serial support registered for generic [ 3.557525] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.560851] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.562357] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.564694] mousedev: PS/2 mouse device common for all mice [ 3.568200] rtc_cmos 00:05: RTC can wake from S4 [ 3.572067] rtc_cmos 00:05: registered as rtc0 [ 3.572138] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.573716] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.580292] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.580737] intel_pstate: CPU model not supported [ 3.586379] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.590588] hid: raw HID events driver (C) Jiri Kosina [ 3.592764] usbcore: registered new interface driver usbhid [ 3.594548] usbhid: USB HID core driver [ 3.596537] drop_monitor: Initializing network drop monitor service [ 3.598765] Initializing XFRM netlink socket [ 3.600580] NET: Registered protocol family 10 [ 3.604037] Segment Routing with IPv6 [ 3.605926] NET: Registered protocol family 17 [ 3.608232] mpls_gso: MPLS GSO support [ 3.614894] RAS: Correctable Errors collector initialized. [ 3.616786] AVX version of gcm_enc/dec engaged. [ 3.618069] AES CTR mode by8 optimization enabled [ 3.698682] sched_clock: Marking stable (3698664234, 0)->(4599230135, -900565901) [ 3.701893] registered taskstats version 1 [ 3.703959] Loading compiled-in X.509 certificates [ 3.707230] zswap: loaded using pool lzo/zbud [ 3.732993] Key type big_key registered [ 3.744908] Key type encrypted registered [ 3.746701] ima: No TPM chip found, activating TPM-bypass! [ 3.748738] ima: Allocated hash algorithm: sha1 [ 3.750121] ima: No architecture policies found [ 3.752143] evm: Initialising EVM extended attributes: [ 3.753956] evm: security.selinux [ 3.754849] evm: security.ima [ 3.755605] evm: security.capability [ 3.757051] evm: HMAC attrs: 0x1 [ 3.759615] rtc_cmos 00:05: setting system clock to 2026-08-22 07:29:38 UTC (1787383778) [ 3.766478] debug: unmapping init [mem 0xffffffff8f603000-0xffffffff8f7fffff] [ 3.769815] debug: unmapping init [mem 0xffffffff8e382000-0xffffffff8e658fff] [ 3.778142] Write protecting the kernel read-only data: 28672k [ 3.781939] debug: unmapping init [mem 0xffffffff8ca03000-0xffffffff8cbfffff] [ 3.785042] debug: unmapping init [mem 0xffffffff8d314000-0xffffffff8d3fffff] [ 3.823817] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.832819] systemd[1]: Detected virtualization kvm. [ 3.834852] systemd[1]: Detected architecture x86-64. [ 3.836962] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.864440] systemd[1]: No hostname configured. [ 3.866434] systemd[1]: Set hostname to . [ 3.868785] random: systemd: uninitialized urandom read (16 bytes read) [ 3.871328] systemd[1]: Initializing machine ID from random generator. [ 4.013129] random: systemd: uninitialized urandom read (16 bytes read) [ 4.015635] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.020341] random: systemd: uninitialized urandom read (16 bytes read) [ 4.022831] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.027747] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.690186] device-mapper: uevent: version 1.0.3 [ 4.692453] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.442773] virtio_net virtio0 ens2: renamed from eth0 [ 5.450747] random: fast init done [ 5.535544] scsi host0: ata_piix [ 5.557738] scsi host1: ata_piix [ 5.559885] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.562826] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.850718] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.262416] random: crng init done [ 10.265367] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.650583] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.807418] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.086078] SELinux: Disabled at runtime. [ 12.152507] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.162159] systemd[1]: Detected virtualization kvm. [ 12.164108] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.655975] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.659691] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.664802] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.672332] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.676226] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.683147] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.690191] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ 12.855920] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.109395] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.371441] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.410979] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.513879] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.523665] EDAC sbridge: Ver: 1.1.2 [ 15.035135] Key type dns_resolver registered [ 15.346161] NFS: Registering the id_resolver key type [ 15.351072] Key type id_resolver registered [ 15.352810] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg619-server login: [ 48.467355] libcfs: loading out-of-tree module taints kernel. [ 48.504642] Key type ._llcrypt registered [ 48.508465] Key type .llcrypt registered [ 48.632116] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 65.387893] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 66.757230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 66.767364] alg: No test for adler32 (adler32-zlib) [ 68.032368] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 68.920686] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 70.655163] Key type lgssc registered [ 72.470459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 89.992837] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 94.919816] hrtimer: interrupt took 5806415 ns [ 128.470125] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 144.971613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 145.024085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 146.230146] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 146.269281] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 146.353572] Lustre: lustre-MDT0000: new disk, initializing [ 146.426070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 146.438852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 151.760316] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 164.844283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 164.934872] Lustre: 6504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 164.965867] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 164.969141] Lustre: Skipped 1 previous similar message [ 165.063799] Lustre: lustre-MDT0001: new disk, initializing [ 165.146911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 165.174625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 165.192935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 169.469545] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 174.420833] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 183.522152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 183.812704] Lustre: lustre-OST0000: new disk, initializing [ 183.818544] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 183.829831] Lustre: 8440:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 183.899246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 188.865700] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.461556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 189.467705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 189.524406] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 200.587806] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 200.704515] Lustre: lustre-OST0001: new disk, initializing [ 200.710129] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 200.719115] Lustre: 9516:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 200.783218] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 206.370606] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 206.926229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 206.937585] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 207.000793] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 218.138409] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 226.824222] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 233.126531] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing check_logdir /tmp/testlogs/ [ 237.979901] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing yml_node [ 242.563726] Lustre: DEBUG MARKER: Client: 2.17.54.113 [ 245.019909] Lustre: DEBUG MARKER: MDS: 2.17.54.113 [ 247.511096] Lustre: DEBUG MARKER: OSS: 2.17.54.113 [ 249.017574] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 03:33:42 EDT 2026 [ 265.553263] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 267.239628] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 283.104084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 283.120536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 283.136064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 284.128703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 284.130025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 284.143530] Lustre: Skipped 3 previous similar messages [ 289.012487] Lustre: server umount lustre-MDT0000 complete [ 292.260734] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384067 with bad export cookie 7637315596442374028 [ 292.261188] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 292.276350] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 294.370755] LustreError: 8448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 294.373961] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 294.377184] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 294.399496] LustreError: 8448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 294.409687] Lustre: Skipped 3 previous similar messages [ 297.376131] LustreError: 6510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 297.378075] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 297.392638] LustreError: 6510:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 297.405894] Lustre: Skipped 2 previous similar messages [ 298.531827] Lustre: server umount lustre-MDT0001 complete [ 302.346071] Lustre: server umount lustre-OST0000 complete [ 305.924679] Lustre: server umount lustre-OST0001 complete [ 312.251504] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 321.303295] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 355.546790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 355.763937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 355.795576] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 355.900139] Lustre: lustre-MDT0000: new disk, initializing [ 355.981221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 355.996773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 359.835394] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 369.747522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 369.822175] Lustre: 15065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 369.864315] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 369.867494] Lustre: Skipped 1 previous similar message [ 369.922887] Lustre: lustre-MDT0001: new disk, initializing [ 369.993799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 370.028481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 370.037531] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 374.275221] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 384.038415] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 390.082743] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 397.859352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 398.123142] Lustre: lustre-OST0000: new disk, initializing [ 398.127991] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 398.140921] Lustre: 17035:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 398.234580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 400.004729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 400.019794] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 400.156911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 404.425901] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 413.750615] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 418.702554] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 418.854933] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 423.511952] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 423.801471] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 431.072259] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 431.095387] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 431.104753] LustreError: Skipped 1 previous similar message [ 431.105405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 431.148396] Lustre: Skipped 1 previous similar message [ 440.287186] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 440.491342] Lustre: server umount lustre-OST0000 complete [ 446.435362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.442850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 446.462901] Lustre: Skipped 1 previous similar message [ 446.488049] Lustre: Skipped 4 previous similar messages [ 451.105913] Lustre: server umount lustre-MDT0000 complete [ 451.552606] LustreError: 15950:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 451.575321] LustreError: 15950:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 455.144772] LustreError: 17034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384229 with bad export cookie 7637315596442375568 [ 455.144772] LustreError: 15060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384229 with bad export cookie 7637315596442375568 [ 455.145753] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 455.159125] LustreError: 15060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 455.442945] Lustre: server umount lustre-MDT0001 complete [ 461.102545] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 03:37:13 (1787384233) [ 479.839076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 480.387496] LustreError: 19911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 480.469870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 484.720679] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 485.874573] LustreError: 19912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 492.185659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 496.589297] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 503.137342] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 508.237527] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 510.624126] Lustre: Failing over lustre-MDT0000 [ 510.917561] Lustre: server umount lustre-MDT0000 complete [ 513.004705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 513.017536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 513.041494] LustreError: 20638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 513.057144] LustreError: 20638:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 529.883244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 529.990249] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 533.872421] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 535.524988] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 535.532787] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 535.578481] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 543.267150] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 549.856768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 549.862942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 549.882281] Lustre: Skipped 2 previous similar messages [ 549.899498] Lustre: Skipped 1 previous similar message [ 553.426662] Lustre: server umount lustre-MDT0000 complete [ 554.988673] LustreError: 19912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 555.015729] LustreError: 19912:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 556.495921] LustreError: 21478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384331 with bad export cookie 7637315596442376982 [ 556.499361] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 556.508441] LustreError: 21478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 556.844581] Lustre: server umount lustre-MDT0001 complete [ 564.946741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 565.500489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 565.510254] Lustre: Skipped 2 previous similar messages [ 569.356401] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 576.775782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 581.091217] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 587.133813] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 591.889538] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 593.869507] Lustre: Failing over lustre-MDT0000 [ 594.087432] Lustre: server umount lustre-MDT0000 complete [ 595.426772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 595.430588] LustreError: 23242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.436781] Lustre: Skipped 1 previous similar message [ 595.446563] LustreError: 23242:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 611.808638] Lustre: 3640:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787384370/real 1787384370] req@ffff976603c6dc00 x1874207800743552/t0(0) o400->MGC192.168.206.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787384386 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 611.834267] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 611.904420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 622.060378] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x69fd32121b7f9ec0 [ 622.077116] LustreError: 25246:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.206.119@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (ffff9766035a8e00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 622.097329] Lustre: MGC192.168.206.119@tcp: Connection restored to 0@lo (at 0@lo) [ 622.110604] Lustre: Skipped 1 previous similar message [ 622.355276] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 626.663552] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 627.681993] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 627.688143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 627.706155] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 635.563906] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 643.044397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 643.051724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 643.053189] Lustre: Skipped 1 previous similar message [ 645.916579] Lustre: server umount lustre-MDT0000 complete [ 650.214202] LustreError: 23228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384424 with bad export cookie 7637315596442377920 [ 650.217047] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 650.222368] LustreError: 23228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 650.901568] Lustre: server umount lustre-MDT0001 complete [ 660.569983] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 03:40:33 (1787384433) [ 678.014591] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 689.068356] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 727.465634] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 737.487575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 737.720442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 737.759270] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 737.841633] Lustre: lustre-MDT0000: new disk, initializing [ 737.914429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 737.918685] Lustre: Skipped 2 previous similar messages [ 737.934758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 742.390255] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 752.021629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 752.121060] Lustre: 29623:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 752.341162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 756.371827] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 764.639239] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 769.121620] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 774.975064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 775.227302] Lustre: 31592:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 776.613690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 776.636870] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 780.986888] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 781.828893] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 781.845193] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 781.848735] Lustre: Skipped 1 previous similar message [ 781.853443] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 789.472978] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 794.635275] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 817.638513] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 817.638768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 817.649861] LustreError: Skipped 1 previous similar message [ 817.655793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 817.666864] Lustre: Skipped 3 previous similar messages [ 817.686438] Lustre: Skipped 1 previous similar message [ 827.359130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 827.552438] Lustre: server umount lustre-OST0000 complete [ 837.457604] Lustre: server umount lustre-MDT0000 complete [ 838.113508] LustreError: 29635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 838.136306] LustreError: 29635:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [ 840.667519] LustreError: 29615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384615 with bad export cookie 7637315596442378326 [ 840.675193] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 840.683594] LustreError: 29615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 841.252576] Lustre: server umount lustre-MDT0001 complete [ 854.065609] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 857.717643] Key type lgssc unregistered [ 858.049937] LNet: 33568:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.058151] LNetError: 33568:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.076700] LNet: Removed LNI 192.168.206.119@tcp [ 859.308142] Key type .llcrypt unregistered [ 859.309815] Key type ._llcrypt unregistered [ 881.906458] Key type ._llcrypt registered [ 881.910983] Key type .llcrypt registered [ 882.013443] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 883.463681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 883.553051] alg: No test for adler32 (adler32-zlib) [ 884.611970] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 884.928125] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 886.560503] Key type lgssc registered [ 887.571261] Lustre: Echo OBD driver; http://www.lustre.org/ [ 894.294959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 894.320837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 895.891468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 899.928664] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 906.213256] LustreError: 34982:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 906.228321] LustreError: 34982:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 908.794093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 909.094991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 913.331700] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 920.386570] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 926.425067] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 932.996775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 933.332371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 934.394237] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 939.477692] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 947.131712] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 961.093536] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 03:45:33 (1787384733) [ 962.906617] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 964.523501] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 970.769923] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 03:45:43 (1787384743) [ 980.839670] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 03:45:53 (1787384753) [ 981.932763] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 993.001076] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 993.007184] Lustre: Skipped 2 previous similar messages [ 994.692287] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 03:46:07 (1787384767) [ 997.904079] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 997.910839] Lustre: Skipped 3 previous similar messages [ 1012.970206] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1012.980555] Lustre: Skipped 6 previous similar messages [ 1015.126537] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 03:46:28 (1787384788) [ 1017.397687] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1020.411544] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1027.694949] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 03:46:40 (1787384800) [ 1029.232701] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1029.237360] Lustre: Skipped 3 previous similar messages [ 1041.350386] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 03:46:54 (1787384814) [ 1057.248542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1057.258802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1057.282428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1058.274721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1058.277319] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1058.291379] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1060.814224] Lustre: server umount lustre-OST0000 complete [ 1067.999988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1068.011063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1068.022599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1068.033232] Lustre: Skipped 1 previous similar message [ 1073.632993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1073.640115] Lustre: Skipped 3 previous similar messages [ 1078.753385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1078.761116] Lustre: Skipped 1 previous similar message [ 1079.263304] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1079.484898] Lustre: server umount lustre-MDT0000 complete [ 1083.094300] LustreError: 38548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384857 with bad export cookie 13484641245426633796 [ 1083.098076] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1083.103336] LustreError: 38548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1083.371325] Lustre: server umount lustre-MDT0001 complete [ 1093.799764] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 1096.470088] Key type lgssc unregistered [ 1096.771146] LNet: 42841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1096.777051] LNetError: 42841:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1096.794618] LNet: Removed LNI 192.168.206.119@tcp [ 1097.651665] Key type .llcrypt unregistered [ 1097.654247] Key type ._llcrypt unregistered [ 1115.173295] Key type ._llcrypt registered [ 1115.175255] Key type .llcrypt registered [ 1115.304059] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 1131.243584] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 1132.370311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1132.443698] alg: No test for adler32 (adler32-zlib) [ 1133.464518] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 1133.707782] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 1135.385573] Key type lgssc registered [ 1136.408933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1173.251586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1173.305821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1174.556283] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1174.592748] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1174.696320] Lustre: lustre-MDT0000: new disk, initializing [ 1174.767490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1174.791451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1180.096388] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1190.794782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1190.915084] Lustre: 45993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1190.987691] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1190.993185] Lustre: Skipped 1 previous similar message [ 1191.076207] Lustre: lustre-MDT0001: new disk, initializing [ 1191.212100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1191.261280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1191.267576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1195.669295] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1204.565493] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1209.660764] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1216.291604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1216.618291] Lustre: lustre-OST0000: new disk, initializing [ 1216.623663] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1216.629147] Lustre: 47961:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1216.714860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1223.057791] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1226.366629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1226.376581] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1226.499499] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1233.001405] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1237.987406] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1238.174427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1243.106575] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1243.364779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1247.211222] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1247.213785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1247.217470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1247.222335] LustreError: Skipped 1 previous similar message [ 1247.251110] Lustre: Skipped 1 previous similar message [ 1252.313552] Lustre: server umount lustre-OST0000 complete [ 1257.441085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1257.445509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1257.480872] Lustre: Skipped 1 previous similar message [ 1257.498121] Lustre: Skipped 2 previous similar messages [ 1262.642681] Lustre: server umount lustre-MDT0000 complete [ 1266.422126] LustreError: 45986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385041 with bad export cookie 6272010379455135785 [ 1266.456715] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1266.990693] Lustre: server umount lustre-MDT0001 complete [ 1277.801969] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 03:50:50 (1787385050) [ 1284.606587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1285.267536] LustreError: 50420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1285.286094] LustreError: 50420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1285.331092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1289.588453] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1290.738486] LustreError: 50421:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1295.843870] LustreError: 50420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1298.577041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1298.974315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1303.774324] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1311.224443] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1316.657323] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1326.253971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1326.809191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1334.555476] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1343.250372] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1348.019125] LustreError: 52951:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1349.168550] Lustre: Modifying parameter general.timeout=1 in log params [ 1351.114749] Lustre: Modifying parameter general.timeout=3 in log params [ 1351.119422] Lustre: Skipped 1 previous similar message [ 1353.594795] Lustre: Modifying parameter general.timeout=6 in log params [ 1353.599774] Lustre: Skipped 2 previous similar messages [ 1357.852485] Lustre: Modifying parameter general.timeout=10 in log params [ 1357.856591] Lustre: Skipped 3 previous similar messages [ 1366.126987] Lustre: Modifying parameter general.timeout=19 in log params [ 1366.131293] Lustre: Skipped 8 previous similar messages [ 1382.971256] Lustre: Modifying parameter general.timeout=39 in log params [ 1382.976202] Lustre: Skipped 19 previous similar messages [ 1415.260815] Lustre: Modifying parameter general.timeout=75 in log params [ 1415.271624] Lustre: Skipped 35 previous similar messages [ 1468.821635] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 03:54:01 (1787385241) [ 1471.693056] Lustre: 59885:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1473.453764] Lustre: 59993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1477.095622] Lustre: 60191:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1477.109564] Lustre: 60191:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1479.535283] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 in log params [ 1479.542913] Lustre: Skipped 61 previous similar messages [ 1507.917868] Lustre: 61765:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1507.931562] Lustre: 61765:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1507.971221] LustreError: 61765:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1512.662753] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 03:54:45 (1787385285) [ 1524.706451] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1524.714627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1524.728091] Lustre: Skipped 1 previous similar message [ 1529.313544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1529.327380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1529.360316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1529.363750] Lustre: Skipped 1 previous similar message [ 1530.722933] Lustre: server umount lustre-MDT0000 complete [ 1534.314192] LustreError: 50400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385309 with bad export cookie 6272010379455136674 [ 1534.316422] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1534.329320] LustreError: 50400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1534.652339] Lustre: server umount lustre-MDT0001 complete [ 1549.401816] Lustre: server umount lustre-OST0000 complete [ 1577.220746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1577.595053] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1577.906158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1582.330506] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1590.303381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1590.401982] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1590.447416] Lustre: 64296:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1595.540550] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1602.526381] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1607.461966] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1612.767561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1612.920723] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1613.129402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1613.133073] Lustre: Skipped 1 previous similar message [ 1618.405683] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1625.482831] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1627.074725] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1627.081995] Lustre: Skipped 16 previous similar messages [ 1627.132666] Lustre: 66861:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1636.834614] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1636.847681] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1636.878194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1636.886554] Lustre: Skipped 2 previous similar messages [ 1639.867562] Lustre: server umount lustre-OST0000 complete [ 1647.073910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1647.084704] LustreError: Skipped 1 previous similar message [ 1647.092429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1647.107908] Lustre: Skipped 1 previous similar message [ 1647.113814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1647.119713] Lustre: Skipped 2 previous similar messages [ 1649.695831] Lustre: server umount lustre-MDT0000 complete [ 1652.715135] LustreError: 64308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1652.737017] LustreError: 64308:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1653.715964] LustreError: 64289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385428 with bad export cookie 6272010379455139628 [ 1653.718595] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1653.725300] LustreError: 64289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1654.262040] Lustre: server umount lustre-MDT0001 complete [ 1664.854418] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 1667.211618] Key type lgssc unregistered [ 1667.463778] LNet: 68071:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1667.471656] LNetError: 68071:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1667.504912] LNet: Removed LNI 192.168.206.119@tcp [ 1668.467583] Key type .llcrypt unregistered [ 1668.471120] Key type ._llcrypt unregistered [ 1691.121964] Key type ._llcrypt registered [ 1691.131343] Key type .llcrypt registered [ 1691.277500] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 1692.300577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1692.414303] alg: No test for adler32 (adler32-zlib) [ 1693.534020] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 1693.808294] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 1695.503200] Key type lgssc registered [ 1696.766231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1703.757278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1703.775299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1705.207937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1709.705944] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1714.659786] LustreError: 69486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1714.679320] LustreError: 69486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1718.608623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1719.511801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1724.226458] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1731.190104] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1737.255660] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1745.230866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1745.984332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1754.507481] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1761.972255] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1769.160699] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1778.666273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1778.669940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1778.694872] Lustre: Skipped 1 previous similar message [ 1778.719782] Lustre: Skipped 2 previous similar messages [ 1783.795632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1783.808612] Lustre: Skipped 2 previous similar messages [ 1783.892884] Lustre: server umount lustre-MDT0000 complete [ 1787.707231] LustreError: 72439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385562 with bad export cookie 13816911509536926956 [ 1787.728462] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1788.067734] Lustre: server umount lustre-MDT0001 complete [ 1801.639627] Lustre: server umount lustre-OST0000 complete [ 1829.717884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1829.971827] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1829.994121] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1830.186955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1833.934143] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1842.232552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1842.297171] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1842.320426] Lustre: 74356:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1842.717499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1846.795477] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1853.386772] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1858.598184] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1865.627716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1865.767882] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1866.058733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1871.765967] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1880.483356] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1882.011929] Lustre: 76929:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1883.950878] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1890.281024] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1890.293896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1890.322297] Lustre: Skipped 1 previous similar message [ 1890.337824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1895.906361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1895.915767] Lustre: Skipped 1 previous similar message [ 1901.031489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1901.044604] Lustre: Skipped 1 previous similar message [ 1903.071418] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1903.318603] Lustre: server umount lustre-OST0000 complete [ 1909.223563] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1909.237649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1909.266867] Lustre: Skipped 1 previous similar message [ 1909.272706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1909.279863] Lustre: Skipped 1 previous similar message [ 1911.267266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1913.989970] Lustre: server umount lustre-MDT0000 complete [ 1916.385656] LustreError: 77102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1916.395088] LustreError: 77102:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1916.947469] LustreError: 76931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385691 with bad export cookie 13816911509536927810 [ 1916.954957] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1916.958702] LustreError: 76931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1917.212416] Lustre: server umount lustre-MDT0001 complete [ 1927.545750] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 1930.233181] Key type lgssc unregistered [ 1930.579566] LNet: 78158:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1930.588711] LNetError: 78158:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1930.610980] LNet: Removed LNI 192.168.206.119@tcp [ 1931.529333] Key type .llcrypt unregistered [ 1931.535452] Key type ._llcrypt unregistered [ 1957.048850] Key type ._llcrypt registered [ 1957.051842] Key type .llcrypt registered [ 1957.143515] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 1958.387082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1958.417913] alg: No test for adler32 (adler32-zlib) [ 1959.445233] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 1959.712230] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 1961.399390] Key type lgssc registered [ 1962.779837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1970.862565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1970.878846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1972.300566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1977.465260] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1981.418358] LustreError: 79576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1981.440213] LustreError: 79576:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1986.164959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1986.887255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1991.637205] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1998.716172] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2004.628912] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2012.522713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2013.215407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2019.939861] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2027.739204] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2037.394597] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2038.410206] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2040.317141] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2040.320807] Lustre: Skipped 1 previous similar message [ 2043.199452] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2043.206740] Lustre: Skipped 2 previous similar messages [ 2048.066486] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2048.073421] Lustre: Skipped 4 previous similar messages [ 2056.638979] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2056.642582] Lustre: Skipped 8 previous similar messages [ 2072.840939] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2072.845965] Lustre: Skipped 17 previous similar messages [ 2104.873760] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2104.881654] Lustre: Skipped 34 previous similar messages [ 2138.957749] Lustre: 87676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2141.815363] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 04:05:14 (1787385914) [ 2143.827581] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2145.964341] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 04:05:18 (1787385918) [ 2154.464065] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2154.465744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2154.471572] LustreError: Skipped 1 previous similar message [ 2154.474412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2154.485519] Lustre: Skipped 1 previous similar message [ 2154.506254] Lustre: Skipped 1 previous similar message [ 2159.592037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2159.601327] Lustre: Skipped 1 previous similar message [ 2164.711568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2164.723745] Lustre: Skipped 1 previous similar message [ 2165.215422] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2165.498373] Lustre: server umount lustre-OST0000 complete [ 2169.829783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2169.845519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2169.850063] Lustre: Skipped 1 previous similar message [ 2174.949241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2174.959425] Lustre: Skipped 2 previous similar messages [ 2175.848932] Lustre: server umount lustre-MDT0000 complete [ 2180.006620] LustreError: 79556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385954 with bad export cookie 18137708314166388574 [ 2180.021108] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2180.394596] Lustre: server umount lustre-MDT0001 complete [ 2192.420533] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 2195.644743] Key type lgssc unregistered [ 2196.136576] LNet: 88926:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2196.148885] LNetError: 88926:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2196.170163] LNet: Removed LNI 192.168.206.119@tcp [ 2197.091171] Key type .llcrypt unregistered [ 2197.092887] Key type ._llcrypt unregistered [ 2202.694583] Key type ._llcrypt registered [ 2202.698687] Key type .llcrypt registered [ 2202.904296] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2207.294591] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules [ 2208.567878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2208.626934] alg: No test for adler32 (adler32-zlib) [ 2209.910518] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 2210.116038] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 2211.790313] Key type lgssc registered [ 2213.242160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2214.327401] LustreError: 90130:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2232.682551] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 2236.937078] LustreError: 90130:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2243.282749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2243.309289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2244.740322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2248.273900] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2254.374402] LustreError: 91157:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2254.400210] LustreError: 91157:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2256.007248] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 04:07:08 (1787386028) [ 2259.426352] LustreError: 91158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2260.632651] LustreError: 91189:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2260.648664] LustreError: 91189:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2260.654394] LustreError: 91189:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 2261.006186] Lustre: server umount lustre-MDT0000 complete [ 2274.822966] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 2277.788859] Key type lgssc unregistered [ 2278.042569] LNet: 92684:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2278.065990] LNetError: 92684:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2278.095205] LNet: Removed LNI 192.168.206.119@tcp [ 2279.021373] Key type .llcrypt unregistered [ 2279.028733] Key type ._llcrypt unregistered [ 2296.551300] Key type ._llcrypt registered [ 2296.553108] Key type .llcrypt registered [ 2296.753791] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 2298.413543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2298.490187] alg: No test for adler32 (adler32-zlib) [ 2299.750793] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 2300.079972] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 2301.783173] Key type lgssc registered [ 2303.539704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2312.800255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2312.819957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2314.262643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2319.846286] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2323.428047] LustreError: 93904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2323.457394] LustreError: 93904:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2328.566598] LustreError: 93903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2329.090907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2329.684729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2334.130905] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2341.463599] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2346.737988] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2352.661849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2353.251428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2359.016279] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2366.495331] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2416.270485] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 04:09:49 (1787386189) [ 2418.600810] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2421.021375] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 04:09:53 (1787386193) [ 2426.849509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2426.865099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2426.892688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2430.433952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2430.439683] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2430.466808] Lustre: Skipped 1 previous similar message [ 2432.175894] Lustre: server umount lustre-MDT0000 complete [ 2435.552884] LustreError: 94634:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2435.577175] LustreError: 94634:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2435.999865] LustreError: 93884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386210 with bad export cookie 17183294542831609933 [ 2436.030418] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2436.355831] Lustre: server umount lustre-MDT0001 complete [ 2451.313796] Lustre: server umount lustre-OST0000 complete [ 2463.092753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2463.544086] LustreError: 98173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2463.727331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2468.016776] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2468.835165] LustreError: 98174:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2476.739506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2477.501553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2482.878906] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2490.210706] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2495.834992] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2510.769569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2511.019912] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2511.027911] LustreError: 98163:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2511.053450] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2511.059994] LustreError: 100258:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2511.067833] LustreError: 100258:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2511.076217] LustreError: 100258:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2511.145477] Lustre: server umount lustre-OST0000 complete [ 2511.148702] LustreError: 100258:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2520.385881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2520.707127] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2520.717397] LustreError: 98163:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2520.756929] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2520.772298] LustreError: 100723:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2520.788957] LustreError: 100723:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2520.793642] LustreError: 100723:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2520.896837] Lustre: server umount lustre-OST0000 complete [ 2520.904284] LustreError: 100723:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2530.603240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2530.752862] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2530.756964] Lustre: Found index 0 for lustre-OST0000, updating log [ 2530.762655] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2530.836385] Lustre: lustre-OST0000: new disk, initializing [ 2530.841831] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2530.849129] Lustre: 101260:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2531.100687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2536.847244] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2540.153946] Lustre: Failing over lustre-OST0000 [ 2540.319831] Lustre: server umount lustre-OST0000 complete [ 2544.615681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2544.635650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2544.645933] Lustre: Skipped 3 previous similar messages [ 2546.731328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2546.737898] Lustre: Skipped 1 previous similar message [ 2551.776861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2551.791569] Lustre: Skipped 1 previous similar message [ 2553.929896] Lustre: server umount lustre-MDT0000 complete [ 2556.901263] LustreError: 98174:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2556.929357] LustreError: 98174:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2558.302702] LustreError: 98154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386333 with bad export cookie 17183294542831612677 [ 2558.305370] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2558.311894] LustreError: 98154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2558.925340] Lustre: server umount lustre-MDT0001 complete [ 2568.424558] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 04:12:20 (1787386340) [ 2585.624630] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 2596.665278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2597.251856] LustreError: 103605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2597.349862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2603.049975] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2612.641954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2618.948331] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2622.074563] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2629.026504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2629.730968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2629.735781] Lustre: Skipped 1 previous similar message [ 2629.750575] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2636.531929] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2637.932519] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2637.937404] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2637.958911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2643.118260] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:04 [ 2643.144342] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2643.163406] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2643.165907] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2643.181292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2644.177142] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2644.260210] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2644.340610] Lustre: lustre-OST0001: new disk, initializing [ 2644.343715] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2644.348774] Lustre: 105925:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2650.481917] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2653.247310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2653.256662] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2653.344363] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2657.038781] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2663.633253] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2667.903574] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.19@tcp (stopping) [ 2669.045773] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2669.066726] Lustre: Skipped 2 previous similar messages [ 2674.084268] Lustre: server umount lustre-MDT0001 complete [ 2674.146382] LustreError: 103605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2674.160714] LustreError: 103605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2682.811180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2682.943082] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2682.946458] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2682.971207] Lustre: 103593:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2684.340664] LustreError: 107463:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2684.350641] LustreError: 107463:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2684.357914] LustreError: 103597:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2684.464493] LustreError: 107477:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.119@tcp: cfg command failed: rc = -17 [ 2684.483334] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2684.483334] [ 2687.748515] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2689.008994] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2696.990513] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 04:14:29 (1787386469) [ 2712.660209] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 2727.933918] Lustre: 109538:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2749.384785] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2753.425841] Lustre: 110673:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2840.543914] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2840.548649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2840.556977] Lustre: Skipped 3 previous similar messages [ 2840.567709] Lustre: Skipped 7 previous similar messages [ 2846.035750] Lustre: server umount lustre-MDT0000 complete [ 2849.191478] LustreError: 106700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386623 with bad export cookie 17183294542831613678 [ 2849.191695] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2849.205519] LustreError: 106700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2849.490793] Lustre: server umount lustre-MDT0001 complete [ 2863.738436] Lustre: server umount lustre-OST0000 complete [ 2866.656121] Lustre: 93215:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787386625/real 1787386625] req@ffff97670ee0b800 x1874210140289408/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787386641 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2866.739258] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2866.756319] Lustre: Skipped 1 previous similar message [ 2867.789394] Lustre: server umount lustre-OST0001 complete [ 2873.737395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2882.473699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2890.593715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2900.738343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2924.230164] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 2935.789286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2935.818873] Lustre: lustre-MDT0000: reset Object Index mappings [ 2936.392523] LustreError: 114620:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2936.440276] LustreError: 114620:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2936.520266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2936.525822] Lustre: Skipped 2 previous similar messages [ 2940.952462] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2950.884505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2950.915798] Lustre: lustre-MDT0001: reset Object Index mappings [ 2957.031753] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2960.517884] Lustre: 115769:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2967.850453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2975.784936] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2976.846692] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 2982.934348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 2986.017158] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2992.144973] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2992.147509] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 2993.366486] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 3001.514492] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3005.516218] Lustre: 117646:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3061.175771] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 04:20:33 (1787386833) [ 3089.379733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3089.401648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3089.408379] Lustre: Skipped 7 previous similar messages [ 3102.176036] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3102.422218] Lustre: server umount lustre-MDT0000 complete [ 3104.752157] LustreError: 114621:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3104.775320] LustreError: 114621:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 3105.906964] LustreError: 114602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386880 with bad export cookie 17183294542831660746 [ 3105.911937] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3105.917635] LustreError: 114602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3112.508702] Lustre: server umount lustre-MDT0001 complete [ 3122.637751] Lustre: server umount lustre-OST0000 complete [ 3132.844165] Lustre: server umount lustre-OST0001 complete [ 3139.682202] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 3149.834657] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 3190.742129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3191.066121] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3191.169144] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3191.277983] Lustre: lustre-MDT0000: new disk, initializing [ 3191.388255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3191.395725] Lustre: Skipped 3 previous similar messages [ 3191.417418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3196.543615] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 3205.419175] Lustre: server umount lustre-MDT0000 complete [ 3216.830036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3217.337619] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3217.389659] Lustre: 121918:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3217.427614] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3223.042984] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 3229.003150] Lustre: server umount lustre-MDT0000 complete [ 3242.143824] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 04:23:33 (1787387013) [ 3244.141723] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3246.294815] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 04:23:38 (1787387018) [ 3265.678891] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 3279.633104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3286.751398] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 3297.867279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3298.082557] Lustre: 123967:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3298.195429] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3298.204290] Lustre: Skipped 1 previous similar message [ 3298.372512] Lustre: lustre-MDT0001: new disk, initializing [ 3298.572113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3298.591193] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3304.164838] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 3309.548255] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3316.127854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3316.455623] Lustre: lustre-OST0000: new disk, initializing [ 3316.459839] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3316.482974] Lustre: 125547:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3317.649736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3317.668332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3317.849175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3323.325318] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 3335.837748] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3336.124738] Lustre: lustre-OST0001: new disk, initializing [ 3336.128811] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3336.135084] Lustre: 126420:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3337.881435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3337.898772] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3338.265924] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3344.681903] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 3354.909119] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3358.555399] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3750.506277] Lustre: 132844:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3750.518125] Lustre: 132844:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3750.525749] Lustre: 132844:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3750.538512] Lustre: 132844:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3750.557506] Lustre: 132844:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3750.578118] Lustre: 132844:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4065.678696] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 04:37:18 (1787387838) [ 4080.615408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4080.619620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4080.628738] Lustre: Skipped 6 previous similar messages [ 4080.636772] Lustre: Skipped 13 previous similar messages [ 4082.857879] Lustre: server umount lustre-MDT0000 complete [ 4085.735475] LustreError: 124763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4085.754296] LustreError: 124763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4086.187985] LustreError: 123959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387860 with bad export cookie 17183294542831729780 [ 4086.205056] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4086.212014] LustreError: 123959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4090.850561] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4090.857350] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4090.866233] Lustre: Skipped 3 previous similar messages [ 4090.875865] Lustre: Skipped 3 previous similar messages [ 4092.650706] Lustre: server umount lustre-MDT0001 complete [ 4102.322264] Lustre: server umount lustre-OST0000 complete [ 4111.885562] Lustre: server umount lustre-OST0001 complete [ 4117.703175] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 4129.103759] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 4166.302270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4166.745396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4166.831574] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4166.951207] Lustre: lustre-MDT0000: new disk, initializing [ 4167.042588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4167.053939] Lustre: Skipped 5 previous similar messages [ 4167.078374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4171.341724] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 4181.344285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4181.515948] Lustre: 135900:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4181.568245] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4181.571926] Lustre: Skipped 1 previous similar message [ 4181.633934] Lustre: lustre-MDT0001: new disk, initializing [ 4181.771310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4181.785965] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4186.105592] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 4194.541790] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4199.461363] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4205.409690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4205.609424] Lustre: lustre-OST0000: new disk, initializing [ 4205.613451] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4205.620296] Lustre: 137865:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4205.707446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4205.713059] Lustre: Skipped 1 previous similar message [ 4207.377248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4207.383188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4207.524900] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4211.207556] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 4220.050761] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4230.413051] Lustre: lustre-MDD0000: changelog on [ 4233.412474] Lustre: lustre-MDD0001: changelog on [ 6095.447073] Lustre: lustre-MDD0001: changelog off [ 6097.186915] Lustre: 135906:0:(llog_cat.c:984:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6098.008320] Lustre: lustre-MDD0000: changelog off [ 6107.109902] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6107.110237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6107.148850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6107.160321] Lustre: Skipped 3 previous similar messages [ 6109.776463] Lustre: server umount lustre-OST0000 complete [ 6117.345097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6117.352255] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6117.357577] LustreError: Skipped 1 previous similar message [ 6117.358818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6117.397803] Lustre: Skipped 2 previous similar messages [ 6119.901270] Lustre: server umount lustre-MDT0000 complete [ 6122.469385] LustreError: 136791:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6122.488290] LustreError: 136791:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6122.766377] LustreError: 135891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389897 with bad export cookie 17183294542832552476 [ 6122.773581] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6122.776804] LustreError: 135891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6123.115749] Lustre: server umount lustre-MDT0001 complete [ 6133.733962] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 6136.213836] Key type lgssc unregistered [ 6136.462563] LNet: 147021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6136.470832] LNetError: 147021:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6136.500828] LNet: Removed LNI 192.168.206.119@tcp [ 6137.193371] Key type .llcrypt unregistered [ 6137.196656] Key type ._llcrypt unregistered [ 6155.314884] Key type ._llcrypt registered [ 6155.317258] Key type .llcrypt registered [ 6155.401889] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 6170.654704] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 6171.799566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6171.844974] alg: No test for adler32 (adler32-zlib) [ 6172.859287] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 6173.075334] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 6174.727243] Key type lgssc registered [ 6175.960965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6210.630125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6210.691125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6211.947974] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6211.975678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6212.023261] Lustre: lustre-MDT0000: new disk, initializing [ 6212.076364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6212.089728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6215.910699] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6224.350383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6224.448867] Lustre: 150222:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6224.511738] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6224.518080] Lustre: Skipped 1 previous similar message [ 6224.600922] Lustre: lustre-MDT0001: new disk, initializing [ 6224.670045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6224.693386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6224.702743] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6228.405351] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6236.980204] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6242.250372] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6248.502043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6248.804583] Lustre: lustre-OST0000: new disk, initializing [ 6248.810788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6248.822197] Lustre: 152186:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6248.918373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6255.077763] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6258.227450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6258.252383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6258.371555] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6263.922924] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6270.266887] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_module obdecho/obdecho [ 6272.002938] LustreError: 153189:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6272.008888] LustreError: 153189:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6273.602876] LustreError: 153285:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 6286.307474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6286.317841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6286.332767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6289.382470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6289.394733] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6289.410329] Lustre: Skipped 1 previous similar message [ 6291.606601] Lustre: server umount lustre-MDT0000 complete [ 6294.972446] LustreError: 150212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390069 with bad export cookie 11138481488738055247 [ 6294.983527] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6294.984612] LustreError: 150212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6295.302953] Lustre: server umount lustre-MDT0001 complete [ 6309.192732] Lustre: server umount lustre-OST0000 complete [ 6331.560377] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 6343.449177] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 6381.810296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6382.006406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6382.028586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6382.111332] Lustre: lustre-MDT0000: new disk, initializing [ 6382.175195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6382.203191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6385.782851] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6395.567457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6395.655641] Lustre: 157002:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6395.698700] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6395.702478] Lustre: Skipped 1 previous similar message [ 6395.755752] Lustre: lustre-MDT0001: new disk, initializing [ 6395.834053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6395.879047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6395.883885] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6399.550949] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6409.157482] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6415.312723] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6422.186845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6422.493640] Lustre: lustre-OST0000: new disk, initializing [ 6422.499248] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6422.512249] Lustre: 158972:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6422.640709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6429.010895] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6429.751537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6429.759844] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6429.917181] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6439.806633] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6445.488838] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6445.725777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6450.126738] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6450.364680] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6455.783907] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6455.787755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6455.788330] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6455.796457] Lustre: Skipped 1 previous similar message [ 6455.826388] Lustre: Skipped 3 previous similar messages [ 6458.470610] Lustre: server umount lustre-OST0000 complete [ 6462.432021] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6462.440154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6462.452116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6468.824377] Lustre: server umount lustre-MDT0000 complete [ 6471.137821] LustreError: 157008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6471.160476] LustreError: 157008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6472.812955] LustreError: 156995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390247 with bad export cookie 11138481488738056206 [ 6472.820890] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6472.824323] LustreError: 156995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6473.200295] Lustre: server umount lustre-MDT0001 complete [ 6477.056658] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 05:17:29 (1787390249) [ 6479.022363] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6481.202414] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 05:17:33 (1787390253) [ 6489.084807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6489.486720] LustreError: 161385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6489.561690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6494.340258] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6494.688377] LustreError: 161386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6499.817542] LustreError: 161385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6503.836820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6508.677659] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6516.288056] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6522.716340] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6529.722366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6530.014977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6530.021986] Lustre: Skipped 1 previous similar message [ 6536.724818] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6545.450247] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6552.285361] Lustre: server umount lustre-MDT0000 complete [ 6555.618046] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6555.624338] LustreError: 161380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6555.627475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6555.651747] Lustre: Skipped 3 previous similar messages [ 6555.695176] LustreError: 161380:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6556.057638] LustreError: 161367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390330 with bad export cookie 11138481488738057088 [ 6556.070865] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6556.078428] LustreError: 161367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6556.543291] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.19@tcp (stopping) [ 6556.549877] Lustre: Skipped 2 previous similar messages [ 6562.502434] Lustre: server umount lustre-MDT0001 complete [ 6569.652078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6582.571779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6582.905728] LustreError: 165224:0:(llog.c:1655:llog_backup()) MGC192.168.206.119@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6582.915151] Lustre: 165224:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.119@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6585.312170] LustreError: 165229:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6585.322911] LustreError: 165229:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6602.602519] LustreError: 165229:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6602.634888] LustreError: 165229:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 6636.001386] LustreError: 163215:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6636.017924] LustreError: 163215:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 6643.682444] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6643.689686] Lustre: 165224:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.119@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6684.693152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6689.767377] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6691.496400] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6700.004586] LustreError: 163216:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6700.027611] LustreError: 163216:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 42 previous similar messages [ 6702.088541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6711.268758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6730.894168] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6731.250224] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6738.515398] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6740.193516] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6745.676556] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6747.445430] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6767.074215] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6767.080498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6767.109081] Lustre: Skipped 2 previous similar messages [ 6767.113151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6767.116302] Lustre: Skipped 5 previous similar messages [ 6771.018825] Lustre: server umount lustre-MDT0000 complete [ 6775.221641] LustreError: 164165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390549 with bad export cookie 11138481488738063549 [ 6775.230424] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6775.235750] LustreError: 164165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6775.711568] Lustre: server umount lustre-MDT0001 complete [ 6790.270816] Lustre: server umount lustre-OST0000 complete [ 6812.587407] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 6823.114304] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 6856.780337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6857.027791] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6857.047186] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6857.111303] Lustre: lustre-MDT0000: new disk, initializing [ 6857.203888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6857.209182] Lustre: Skipped 1 previous similar message [ 6857.228643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6861.824381] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6873.673459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6873.887882] Lustre: 170331:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6874.269463] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6879.927898] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6889.925410] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6897.207850] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6905.753411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6906.050516] Lustre: lustre-OST0000: new disk, initializing [ 6906.054400] Lustre: Skipped 1 previous similar message [ 6906.058714] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6906.066244] Lustre: Skipped 2 previous similar messages [ 6906.075578] Lustre: 172302:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6912.097206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6912.130036] Lustre: Skipped 1 previous similar message [ 6912.140265] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6912.233402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6912.674745] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6922.921359] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6928.498238] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6928.703199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6933.478304] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6933.694985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6938.082839] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6938.082854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6938.100496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6938.107439] Lustre: Skipped 2 previous similar messages [ 6938.122057] Lustre: Skipped 1 previous similar message [ 6941.859902] Lustre: server umount lustre-OST0000 complete [ 6960.095124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6960.277347] Lustre: server umount lustre-MDT0000 complete [ 6962.658930] LustreError: 170343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6962.687286] LustreError: 170343:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 27 previous similar messages [ 6963.548619] LustreError: 170322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390738 with bad export cookie 11138481488738080377 [ 6963.555323] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6963.559373] LustreError: 170322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6964.015918] Lustre: server umount lustre-MDT0001 complete [ 6972.489450] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 05:25:45 (1787390745) [ 6978.528783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6983.014281] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6990.954429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6995.806138] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7002.888636] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7008.429354] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7015.054934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7021.871774] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7029.149520] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7038.002603] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 05:26:51 (1787390811) [ 7042.022133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7042.030779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7042.044649] Lustre: Skipped 2 previous similar messages [ 7042.051758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7042.056988] Lustre: Skipped 6 previous similar messages [ 7047.467778] Lustre: server umount lustre-OST0000 complete [ 7065.055200] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7065.241197] Lustre: server umount lustre-MDT0000 complete [ 7068.576461] LustreError: 174695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390843 with bad export cookie 11138481488738081259 [ 7068.593032] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7068.884040] Lustre: server umount lustre-MDT0001 complete [ 7079.280724] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 7081.963861] Key type lgssc unregistered [ 7082.265037] LNet: 178443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7082.273357] LNetError: 178443:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7082.292842] LNet: Removed LNI 192.168.206.119@tcp [ 7083.098696] Key type .llcrypt unregistered [ 7083.103596] Key type ._llcrypt unregistered [ 7099.233554] Key type ._llcrypt registered [ 7099.237825] Key type .llcrypt registered [ 7099.333881] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 7100.002742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7100.146588] alg: No test for adler32 (adler32-zlib) [ 7101.156976] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 7101.361099] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 7103.031277] Key type lgssc registered [ 7104.126694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7110.577403] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7110.614904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7126.178640] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7131.295303] LustreError: 179688:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.119@tcp: failed processing log, type 4: rc = -110 [ 7136.416502] LustreError: 179688:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7136.427745] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7136.431640] LustreError: MGC192.168.206.119@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7136.440454] LustreError: 179688:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7136.446876] LustreError: 179688:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7136.455334] LustreError: 179688:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7136.575746] Lustre: server umount lustre-OST0000 complete [ 7136.577876] LustreError: 179688:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7144.455545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7145.913566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7150.169787] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7156.205604] LustreError: 180187:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7156.228504] LustreError: 180187:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7158.246355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7158.618150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7163.827217] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7172.220273] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7177.622366] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7184.037398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7184.232806] LustreError: 182002:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7184.241638] Lustre: 182002:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.119@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7184.430557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7190.564177] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7197.523136] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7199.729121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7199.738415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7199.757987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7200.741212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7200.747330] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7200.771309] Lustre: Skipped 1 previous similar message [ 7203.875443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7203.879137] Lustre: Skipped 2 previous similar messages [ 7206.249733] Lustre: server umount lustre-MDT0000 complete [ 7208.935581] LustreError: 180914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7208.967495] LustreError: 180914:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7210.233597] LustreError: 182031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390984 with bad export cookie 10321368661247913202 [ 7210.240405] LustreError: 182031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7210.241599] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7212.193330] LustreError: 180916:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009bc90d3b x1874215175221632/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7212.349190] Lustre: server umount lustre-MDT0001 complete [ 7226.222593] Lustre: server umount lustre-OST0000 complete [ 7236.954264] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 7246.783646] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 7288.631808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7289.148898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7289.207422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7289.313430] Lustre: lustre-MDT0000: new disk, initializing [ 7289.464959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7289.509450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7294.432740] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7304.135797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7304.262558] Lustre: 185530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7304.315112] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7304.318959] Lustre: Skipped 1 previous similar message [ 7304.380936] Lustre: lustre-MDT0001: new disk, initializing [ 7304.460093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7304.489387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7304.503059] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7308.901892] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7319.264699] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7325.479559] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7333.553059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7333.824746] Lustre: lustre-OST0000: new disk, initializing [ 7333.829088] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7333.836135] Lustre: 187499:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7333.957747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7340.947515] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7342.175950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7342.188847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7342.340362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7351.381357] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7356.037713] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7356.261799] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7360.988742] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7361.209728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7368.173542] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7368.174499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7368.179993] LustreError: Skipped 1 previous similar message [ 7368.186359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7368.195948] Lustre: Skipped 1 previous similar message [ 7368.219245] Lustre: Skipped 2 previous similar messages [ 7373.286276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7373.291783] Lustre: Skipped 1 previous similar message [ 7377.378721] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7377.541572] Lustre: server umount lustre-OST0000 complete [ 7381.473988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7381.479674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7381.500113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7395.295116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7395.531895] Lustre: server umount lustre-MDT0000 complete [ 7398.792656] LustreError: 185522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391173 with bad export cookie 10321368661247913923 [ 7398.801034] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7398.801783] LustreError: 185522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7399.077180] Lustre: server umount lustre-MDT0001 complete [ 7406.937069] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 05:32:59 (1787391179) [ 7421.699315] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 7424.023213] Key type lgssc unregistered [ 7424.358415] LNet: 190446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7424.371625] LNetError: 190446:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7424.393923] LNet: Removed LNI 192.168.206.119@tcp [ 7425.177432] Key type .llcrypt unregistered [ 7425.179418] Key type ._llcrypt unregistered [ 7442.367566] Key type ._llcrypt registered [ 7442.369305] Key type .llcrypt registered [ 7442.469916] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 7443.426499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7443.457262] alg: No test for adler32 (adler32-zlib) [ 7444.463459] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 7444.667599] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 7446.320048] Key type lgssc registered [ 7447.411491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7454.639203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7454.661250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7456.080090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7460.499844] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7465.441665] LustreError: 191667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7465.456244] LustreError: 191667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7469.480503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7469.778742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7473.869332] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7481.970597] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7487.703250] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7496.150780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7496.381562] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7496.384189] LustreError: 193531:0:(llog.c:1696:llog_backup()) MGC192.168.206.119@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7496.391237] Lustre: 193531:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.119@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7496.493796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7503.539455] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7510.968305] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7514.531269] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7514.546401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7514.551697] Lustre: Skipped 1 previous similar message [ 7519.716815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7519.726491] Lustre: Skipped 3 previous similar messages [ 7519.887433] Lustre: server umount lustre-MDT0000 complete [ 7523.632433] LustreError: 191648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391298 with bad export cookie 2956553731459162093 [ 7523.653258] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7524.838223] LustreError: 191663:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7524.838506] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7524.838937] LustreError: 192397:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000066083023 x1874215534881920/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7524.925229] Lustre: Skipped 1 previous similar message [ 7525.111297] Lustre: server umount lustre-MDT0001 complete [ 7529.289702] Lustre: server umount lustre-OST0000 complete [ 7539.977166] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 7552.134969] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 7592.344563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7592.624709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7592.666726] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7592.770149] Lustre: lustre-MDT0000: new disk, initializing [ 7592.844710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7592.864357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7598.116336] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7608.757643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7608.871593] Lustre: 197052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7608.915889] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7608.921756] Lustre: Skipped 1 previous similar message [ 7608.930349] LustreError: 197836:0:(llog.c:1696:llog_backup()) MGC192.168.206.119@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7608.938367] LustreError: 197836:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7608.945386] Lustre: 197836:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.119@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7608.978082] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7608.983289] Lustre: Skipped 1 previous similar message [ 7609.050748] Lustre: lustre-MDT0001: new disk, initializing [ 7609.159894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7609.215981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7609.225464] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7614.798437] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7626.776793] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7634.098879] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7641.058527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7641.335443] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7641.338681] Lustre: Skipped 1 previous similar message [ 7641.341537] LustreError: 199001:0:(llog.c:1696:llog_backup()) MGC192.168.206.119@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7641.351966] LustreError: 199001:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7641.366200] Lustre: 199001:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.119@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7641.379043] Lustre: 199001:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7641.415516] Lustre: lustre-OST0000: new disk, initializing [ 7641.419767] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7641.427354] Lustre: 199024:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7641.505340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7648.017081] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7650.372082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7650.388190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7650.539867] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7657.869977] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7662.541075] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7662.743703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7668.026406] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7668.238345] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7671.268522] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7671.270535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7671.278108] LustreError: Skipped 1 previous similar message [ 7671.299430] Lustre: Skipped 1 previous similar message [ 7671.303770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7671.307695] Lustre: Skipped 1 previous similar message [ 7676.416768] Lustre: server umount lustre-OST0000 complete [ 7681.017622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7681.027514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7681.060426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7686.625667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7686.633337] Lustre: Skipped 2 previous similar messages [ 7695.329254] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7695.558285] Lustre: server umount lustre-MDT0000 complete [ 7696.870128] LustreError: 197066:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7696.912163] LustreError: 197066:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7699.735786] LustreError: 197047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391474 with bad export cookie 2956553731459162814 [ 7699.736887] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7699.747562] LustreError: 197047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7700.093683] Lustre: server umount lustre-MDT0001 complete [ 7709.659781] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 05:38:02 (1787391482) [ 7729.321741] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 7741.116148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7741.567056] LustreError: 202152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7741.645166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7746.278320] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7747.043947] LustreError: 202153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7756.079447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7761.253300] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7764.705557] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7772.041709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7779.046546] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7788.602086] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7788.811667] Lustre: lustre-OST0001: new disk, initializing [ 7788.815437] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7788.820770] Lustre: 204459:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7788.902408] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7788.906627] Lustre: Skipped 2 previous similar messages [ 7795.360470] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7795.378689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7795.398144] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7795.563181] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7806.830728] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7818.062545] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7820.075542] Lustre: DEBUG MARKER: ADD OST3 [ 7825.475169] loop: module loaded [ 7825.552487] loop0: detected capacity change from 0 to 204800000 [ 7825.830653] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7825.850289] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7825.869532] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7832.522481] loop0: detected capacity change from 0 to 204800000 [ 7833.247304] Lustre: DEBUG MARKER: STOP OST3 [ 7839.212157] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7839.433233] Lustre: lustre-OST0003: new disk, initializing [ 7839.439754] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7839.451776] Lustre: 206472:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7840.972380] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7841.139232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7841.148451] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7841.199106] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7843.039090] LustreError: 206633:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7843.193685] Lustre: Failing over lustre-OST0003 [ 7843.617751] Lustre: server umount lustre-OST0003 complete [ 7846.285341] LustreError: 203637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7846.305244] LustreError: 203637:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7846.381122] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7846.382643] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7846.392774] Lustre: Skipped 1 previous similar message [ 7846.739982] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7855.423554] Lustre: DEBUG MARKER: START OST3 again [ 7861.910983] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7862.551817] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7862.563840] Lustre: Skipped 1 previous similar message [ 7862.585874] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7863.820385] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7864.443801] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7864.448209] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7873.207226] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7896.379835] Lustre: Failing over lustre-OST0003 [ 7896.534190] Lustre: server umount lustre-OST0003 complete [ 7898.622248] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7898.640673] Lustre: Skipped 1 previous similar message [ 7898.648352] LustreError: 203639:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7898.674603] LustreError: 203639:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 7915.491716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7915.496927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7915.504932] Lustre: Skipped 1 previous similar message [ 7915.509085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7915.514718] Lustre: Skipped 2 previous similar messages [ 7921.989763] Lustre: server umount lustre-MDT0000 complete [ 7926.427787] LustreError: 202132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391701 with bad export cookie 2956553731459163696 [ 7926.432749] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7926.446929] LustreError: 202132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7931.361490] LustreError: 202148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7931.396134] LustreError: 202148:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 7932.898168] Lustre: server umount lustre-MDT0001 complete [ 7937.376358] Lustre: server umount lustre-OST0000 complete [ 7941.361246] Lustre: server umount lustre-OST0001 complete [ 7948.791212] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 7958.450752] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 7998.080741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7998.293523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7998.326673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7998.426643] Lustre: lustre-MDT0000: new disk, initializing [ 7998.515304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7998.532659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8003.076560] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8017.288477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8017.411988] Lustre: 211926:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8017.457350] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8017.460580] Lustre: Skipped 1 previous similar message [ 8017.528333] Lustre: lustre-MDT0001: new disk, initializing [ 8017.654290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8017.676622] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8023.078932] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8033.511514] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8041.113265] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8047.929237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8048.295342] Lustre: 213894:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8049.797283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8049.814233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8049.944746] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8055.299188] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8065.718530] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8070.607488] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8070.951505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8076.719940] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8076.968393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8080.864767] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8080.876789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8080.901485] Lustre: Skipped 5 previous similar messages [ 8080.912919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8080.917276] Lustre: Skipped 8 previous similar messages [ 8093.151211] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8093.356955] Lustre: server umount lustre-OST0000 complete [ 8099.808828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8099.819027] LustreError: Skipped 1 previous similar message [ 8103.534596] Lustre: server umount lustre-MDT0000 complete [ 8106.465238] LustreError: 215234:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8106.497106] LustreError: 215234:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8108.907365] LustreError: 211917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391883 with bad export cookie 2956553731459165992 [ 8108.910449] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8108.916357] LustreError: 211917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8109.452593] Lustre: server umount lustre-MDT0001 complete [ 8114.489972] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 05:44:46 (1787391886) [ 8130.385719] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 8140.527568] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 8177.695919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8178.005167] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8178.047302] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8178.052590] Lustre: Skipped 1 previous similar message [ 8178.143251] Lustre: lustre-MDT0000: new disk, initializing [ 8178.146207] Lustre: Skipped 1 previous similar message [ 8178.249525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8182.256873] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8192.187794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8192.343902] Lustre: 218504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8192.654573] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8196.792476] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8206.586827] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8212.926102] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8219.839367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8220.274579] Lustre: 220473:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8222.212366] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8227.196696] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8237.006637] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8242.512151] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8242.748700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8247.465748] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8247.681244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8252.902845] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8252.905208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8252.921085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8252.921091] Lustre: Skipped 8 previous similar messages [ 8252.976979] Lustre: Skipped 4 previous similar messages [ 8255.847732] Lustre: server umount lustre-OST0000 complete [ 8273.375125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8273.683606] Lustre: server umount lustre-MDT0000 complete [ 8277.497753] LustreError: 218516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8277.514103] LustreError: 218516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8277.705492] LustreError: 219308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392052 with bad export cookie 2956553731459166874 [ 8277.712375] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8277.722542] LustreError: 219308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8278.407599] Lustre: server umount lustre-MDT0001 complete [ 8287.097489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8287.815863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8287.821045] Lustre: Skipped 5 previous similar messages [ 8292.249632] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8300.118969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8305.029254] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8311.845705] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8317.758696] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8324.725715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8331.023850] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8338.079557] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8348.653078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8348.669680] Lustre: Skipped 9 previous similar messages [ 8354.322691] Lustre: server umount lustre-OST0000 complete [ 8364.457469] Lustre: server umount lustre-MDT0000 complete [ 8368.794057] LustreError: 222674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392143 with bad export cookie 2956553731459167756 [ 8368.795878] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8368.808251] LustreError: 222674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8369.218112] Lustre: server umount lustre-MDT0001 complete [ 8475.928046] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 8479.625324] Key type lgssc unregistered [ 8480.022863] LNet: 226355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8480.040802] LNetError: 226355:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8480.059388] LNet: Removed LNI 192.168.206.119@tcp [ 8481.145138] Key type .llcrypt unregistered [ 8481.148381] Key type ._llcrypt unregistered [ 8513.408633] Key type ._llcrypt registered [ 8513.412052] Key type .llcrypt registered [ 8513.532435] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 8531.345991] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 8532.677974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8532.816761] alg: No test for adler32 (adler32-zlib) [ 8533.960624] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 8534.187658] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 8535.880735] Key type lgssc registered [ 8536.973810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8574.644387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8574.675975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8575.921601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8575.960326] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8576.114849] Lustre: lustre-MDT0000: new disk, initializing [ 8576.203160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8576.221988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8581.297692] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8592.773220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8592.921972] Lustre: 229709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8592.962921] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8592.966575] Lustre: Skipped 1 previous similar message [ 8593.041482] Lustre: lustre-MDT0001: new disk, initializing [ 8593.141974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8593.172835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8593.198822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8597.863965] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8608.698669] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8614.892497] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8621.712392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8622.021791] Lustre: lustre-OST0000: new disk, initializing [ 8622.025246] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8622.049112] Lustre: 231678:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8622.137826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8628.228750] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8631.854436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8631.867888] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8631.991653] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8637.238906] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8641.215423] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8641.425499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8645.759735] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8646.045910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8652.769422] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8652.776181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8652.778054] LustreError: Skipped 1 previous similar message [ 8652.784759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8652.792453] Lustre: Skipped 1 previous similar message [ 8654.803801] Lustre: server umount lustre-OST0000 complete [ 8659.944098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8659.956378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8659.975944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8659.985732] Lustre: Skipped 1 previous similar message [ 8663.009107] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8663.009398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8663.064866] Lustre: Skipped 1 previous similar message [ 8665.917622] Lustre: server umount lustre-MDT0000 complete [ 8670.580974] LustreError: 229700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392445 with bad export cookie 2703766615930206890 [ 8670.582783] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8670.600714] LustreError: 229700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8671.138708] Lustre: server umount lustre-MDT0001 complete [ 8689.701653] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 8700.602659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8700.971381] LustreError: 234612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8700.990283] LustreError: 234612:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8701.026493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8705.895431] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8706.528611] LustreError: 234612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8706.554242] LustreError: 234612:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8710.626206] LustreError: 234613:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8715.247672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8715.616374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8719.848604] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8723.062645] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8729.851192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8730.227679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8736.550510] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8745.033214] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8745.231441] Lustre: lustre-OST0001: new disk, initializing [ 8745.236593] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8745.242849] Lustre: 236918:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8745.475944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8745.517636] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8745.620221] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8751.248282] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8761.911341] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8769.129233] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8783.390531] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 05:55:55 (1787392555) [ 8787.425459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8787.439834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8787.456756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8791.815847] Lustre: server umount lustre-MDT0000 complete [ 8793.068242] LustreError: 234608:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8793.086798] LustreError: 234608:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8795.224074] LustreError: 234594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392569 with bad export cookie 2703766615930207772 [ 8795.232822] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8795.239192] LustreError: 234594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8795.676926] Lustre: server umount lustre-MDT0001 complete [ 8810.430139] Lustre: server umount lustre-OST0000 complete [ 8813.356761] Lustre: 227935:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787392572/real 1787392572] req@ffff976736fe1180 x1874216676862976/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787392588 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8813.388897] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8813.400118] Lustre: Skipped 3 previous similar messages [ 8814.814845] Lustre: server umount lustre-OST0001 complete [ 8821.200086] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 8831.305240] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 8868.290211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8868.470751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8868.494645] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8868.607675] Lustre: lustre-MDT0000: new disk, initializing [ 8868.692324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8868.698112] Lustre: Skipped 1 previous similar message [ 8868.712788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8873.215086] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8884.077113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8884.209368] Lustre: 240862:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8884.285474] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8884.288834] Lustre: Skipped 1 previous similar message [ 8884.374789] Lustre: lustre-MDT0001: new disk, initializing [ 8884.530935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8884.551863] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8888.818797] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8898.361768] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8904.153944] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8910.379359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8910.598687] Lustre: lustre-OST0000: new disk, initializing [ 8910.601652] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8910.610660] Lustre: 242830:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8910.706112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8910.711790] Lustre: Skipped 1 previous similar message [ 8912.404138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8912.415362] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8912.494722] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8916.612477] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8925.319458] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8930.461097] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8930.647777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8935.551570] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8935.748962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8937.952080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8937.965530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8937.984076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8937.997422] Lustre: Skipped 4 previous similar messages [ 8944.012731] Lustre: server umount lustre-OST0000 complete [ 8948.195613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8948.202196] Lustre: Skipped 3 previous similar messages [ 8954.465275] Lustre: server umount lustre-MDT0000 complete [ 8958.433496] LustreError: 241739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8958.461283] LustreError: 241739:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8958.679671] LustreError: 240854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392733 with bad export cookie 2703766615930209907 [ 8958.680647] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8958.686615] LustreError: 240854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8959.091410] Lustre: server umount lustre-MDT0001 complete [ 8968.181515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8968.600202] LustreError: 245049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8972.574637] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8981.160395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8981.552378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8981.561896] Lustre: Skipped 1 previous similar message [ 8985.953765] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8993.025970] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8998.108656] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9003.679197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9009.857749] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9017.882544] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9165.806955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9165.818706] LustreError: Skipped 1 previous similar message [ 9165.826935] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9165.840358] Lustre: Skipped 3 previous similar messages [ 9165.858625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9165.871132] Lustre: Skipped 2 previous similar messages [ 9169.281354] Lustre: server umount lustre-OST0000 complete [ 9176.032514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9176.048615] LustreError: Skipped 1 previous similar message [ 9179.503407] Lustre: server umount lustre-MDT0000 complete [ 9182.180150] LustreError: 245049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9182.209663] LustreError: 245049:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9183.618766] LustreError: 245030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392958 with bad export cookie 2703766615930210782 [ 9183.632732] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9183.633963] LustreError: 245030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9184.066464] Lustre: server umount lustre-MDT0001 complete [ 9287.855381] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 9291.038194] Key type lgssc unregistered [ 9291.381196] LNet: 248671:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9291.390353] LNetError: 248671:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9291.417752] LNet: Removed LNI 192.168.206.119@tcp [ 9292.384423] Key type .llcrypt unregistered [ 9292.389112] Key type ._llcrypt unregistered [ 9319.356409] Key type ._llcrypt registered [ 9319.359429] Key type .llcrypt registered [ 9319.465122] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 9336.113894] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 9337.213444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9337.254054] alg: No test for adler32 (adler32-zlib) [ 9338.317692] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 9338.637769] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 9340.384289] Key type lgssc registered [ 9342.141678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9393.695520] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 9405.171492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9405.205408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9406.528987] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9406.574371] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9406.663684] Lustre: lustre-MDT0000: new disk, initializing [ 9406.773869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9406.789764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9411.096197] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9422.898161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9423.043648] Lustre: 252743:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9423.096811] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9423.103522] Lustre: Skipped 1 previous similar message [ 9423.191448] Lustre: lustre-MDT0001: new disk, initializing [ 9423.292834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9423.331486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9423.347539] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9427.426839] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9432.093885] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9438.731306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9439.037575] Lustre: lustre-OST0000: new disk, initializing [ 9439.043911] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9439.049727] Lustre: 254375:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9439.155244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9445.028429] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9448.049979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9448.064083] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9448.185381] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9454.977282] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9455.150102] Lustre: lustre-OST0001: new disk, initializing [ 9455.161308] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9455.178707] Lustre: 255247:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9455.298182] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9456.258346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9456.275598] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9456.442250] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9462.104033] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9473.433202] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9482.083914] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9497.582385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9497.599751] Lustre: Skipped 1 previous similar message [ 9497.605434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9497.624159] Lustre: Skipped 3 previous similar messages [ 9501.980819] Lustre: server umount lustre-MDT0000 complete [ 9505.831345] LustreError: 252735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393280 with bad export cookie 12799082310314005006 [ 9505.832718] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9505.843294] LustreError: 252735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9506.204248] Lustre: server umount lustre-MDT0001 complete [ 9519.759400] Lustre: server umount lustre-OST0000 complete [ 9523.691976] Lustre: 250248:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787393282/real 1787393282] req@ffff976736fbd180 x1874217520936320/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787393298 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9523.775786] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9523.794038] Lustre: Skipped 2 previous similar messages [ 9525.495841] Lustre: server umount lustre-OST0001 complete [ 9539.702449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9551.419646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9561.826194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9573.780308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9594.276110] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 9607.233038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9607.251450] Lustre: lustre-MDT0000: reset Object Index mappings [ 9607.591522] LustreError: 260255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9607.615668] LustreError: 260255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9607.666910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9612.774905] LustreError: 260256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9613.152751] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9617.895573] LustreError: 260255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9623.012790] LustreError: 260256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9624.838995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9624.894740] Lustre: lustre-MDT0001: reset Object Index mappings [ 9625.522744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9631.195505] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9636.674621] Lustre: 261398:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9643.946402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9644.508984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9650.669851] LustreError: 261754:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9651.835953] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9652.725400] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9661.576854] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9667.080967] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9669.034571] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9678.090811] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9682.527147] Lustre: 263272:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9687.528726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9687.541293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9687.550499] Lustre: Skipped 2 previous similar messages [ 9687.572700] Lustre: Skipped 2 previous similar messages [ 9689.449288] Lustre: server umount lustre-MDT0000 complete [ 9692.131334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9692.144548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9692.160888] LustreError: 260989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9692.178610] LustreError: 260989:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9693.692104] LustreError: 260235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393468 with bad export cookie 12799082310314011796 [ 9693.699507] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9693.701274] LustreError: 260235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9693.928907] Lustre: server umount lustre-MDT0001 complete [ 9708.195911] Lustre: server umount lustre-OST0000 complete [ 9722.201198] Lustre: server umount lustre-OST0001 complete [ 9741.575523] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 06:11:54 (1787393514) [ 9756.777955] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 9766.308668] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 9805.414267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9805.666943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9805.710694] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9805.823204] Lustre: lustre-MDT0000: new disk, initializing [ 9805.921782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9805.931524] Lustre: Skipped 1 previous similar message [ 9805.969292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9810.528545] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9821.909084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9822.040255] Lustre: 267526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9822.140566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9822.152839] Lustre: Skipped 1 previous similar message [ 9822.280811] Lustre: lustre-MDT0001: new disk, initializing [ 9822.460369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9822.473802] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9827.370574] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9839.646923] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9846.082845] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9853.045987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9853.276780] Lustre: lustre-OST0000: new disk, initializing [ 9853.280594] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9853.286308] Lustre: 269495:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9854.476953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9854.494143] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9854.659073] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9860.250664] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9870.954022] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9876.698978] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9876.940453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9883.789189] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9883.968138] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9890.278590] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9890.279738] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9890.324410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9893.054661] Lustre: server umount lustre-OST0000 complete [ 9899.489165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9899.501657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9899.532073] Lustre: Skipped 1 previous similar message [ 9899.540271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9899.548263] Lustre: Skipped 1 previous similar message [ 9903.594914] Lustre: server umount lustre-MDT0000 complete [ 9905.634354] LustreError: 267537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9905.672455] LustreError: 267537:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9907.574265] LustreError: 267519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393682 with bad export cookie 12799082310314013889 [ 9907.575725] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9907.588196] LustreError: 267519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9908.186671] Lustre: server umount lustre-MDT0001 complete [ 9932.034654] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 9944.706070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9945.383065] LustreError: 272432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9945.487625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9945.503219] Lustre: Skipped 2 previous similar messages [ 9950.224888] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9958.877551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9963.581678] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9966.573627] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9973.249871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9980.421923] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9990.757493] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9990.913636] Lustre: lustre-OST0001: new disk, initializing [ 9990.924200] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9990.928699] Lustre: 274741:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9992.104448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9992.131868] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9992.329143] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9997.839735] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10008.467208] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10012.454884] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10017.866374] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10018.063947] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10025.965618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10025.981976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10026.001250] Lustre: Skipped 1 previous similar message [10026.012146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10026.027619] Lustre: Skipped 2 previous similar messages [10030.886360] Lustre: server umount lustre-MDT0000 complete [10033.133159] LustreError: 275817:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10033.155890] LustreError: 275817:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10035.061634] LustreError: 274740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393809 with bad export cookie 12799082310314014771 [10035.066815] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10035.080626] LustreError: 274740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10035.752448] Lustre: server umount lustre-MDT0001 complete [10050.202488] Lustre: server umount lustre-OST0000 complete [10054.122115] Lustre: 250250:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787393812/real 1787393812] req@ffff97660c1c9500 x1874217521102976/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787393828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10054.708322] Lustre: server umount lustre-OST0001 complete [10074.204336] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [10083.820609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10084.507292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10084.515118] Lustre: Skipped 3 previous similar messages [10088.692356] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10098.528145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10103.484993] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10106.838762] Lustre: 278941:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10114.187648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10120.877559] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10129.810865] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10129.927383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10135.538201] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10137.377812] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10145.671836] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10149.756378] Lustre: 280815:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10155.071194] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10155.125979] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10165.394099] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 06:18:57 (1787393937) [10181.088151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10181.100769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10181.128709] Lustre: Skipped 4 previous similar messages [10181.140019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10181.150930] Lustre: Skipped 4 previous similar messages [10187.364745] Lustre: server umount lustre-MDT0000 complete [10191.030514] LustreError: 277782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393965 with bad export cookie 12799082310314021568 [10191.044121] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10191.049603] LustreError: 277782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10191.841431] LustreError: 277796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10191.886785] LustreError: 277796:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [10197.505779] Lustre: server umount lustre-MDT0001 complete [10207.613395] Lustre: server umount lustre-OST0000 complete [10211.648172] Lustre: server umount lustre-OST0001 complete [10219.037750] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [10231.114817] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [10282.016960] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [10292.308192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10292.529656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10292.589649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10292.705707] Lustre: lustre-MDT0000: new disk, initializing [10292.827774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10297.593926] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10307.224560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10307.335808] Lustre: 285058:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10307.529356] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10311.418577] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10322.188600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10322.416363] Lustre: 286689:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10323.739329] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10329.658111] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10341.602902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10341.800480] Lustre: 287562:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10341.934358] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10341.940950] Lustre: Skipped 6 previous similar messages [10343.065538] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10343.078276] Lustre: Skipped 1 previous similar message [10343.301711] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10348.982556] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10358.212148] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10362.532631] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10362.540860] Lustre: Skipped 1 previous similar message [10377.361956] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 06:22:30 (1787394150) [10379.232534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10379.243204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10379.257951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10379.261708] Lustre: Skipped 8 previous similar messages [10379.272750] Lustre: Skipped 13 previous similar messages [10381.359828] Lustre: server umount lustre-MDT0000 complete [10385.923778] LustreError: 285049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394160 with bad export cookie 12799082310314028995 [10385.923778] LustreError: 285051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394160 with bad export cookie 12799082310314028995 [10385.925485] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10385.942379] LustreError: 285049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10386.517128] Lustre: server umount lustre-MDT0001 complete [10400.453557] Lustre: server umount lustre-OST0000 complete [10415.670362] Lustre: server umount lustre-OST0001 complete [10424.959418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10431.390881] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10440.612351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10445.596711] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10453.504818] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10459.790093] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10466.475411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10470.956672] LustreError: 291642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10470.974940] LustreError: 291642:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [10473.059871] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10481.352589] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10500.583739] loop1: detected capacity change from 0 to 204800000 [10500.767561] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10500.777016] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10500.789354] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10505.686851] loop1: detected capacity change from 0 to 204800000 [10511.689166] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10511.774565] LustreError: 289798:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs16456: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10511.791422] LustreError: 289798:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs16456 [10511.799079] LustreError: 289798:0:(mgs_handler.c:517:mgs_target_reg()) fs16456-MDT0001: Error checking target index: rc = -13 [10511.815454] LustreError: fs16456-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10511.829614] LustreError: 293529:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10511.847175] LustreError: 293529:0:(tgt_mount.c:2129:server_put_super()) no obd fs16456-MDT0001 [10511.854706] LustreError: 293529:0:(tgt_mount.c:116:server_deregister_mount()) fs16456-MDT0001 not registered [10511.922547] Lustre: server umount fs16456-MDT0001 complete [10511.928266] LustreError: 293529:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10521.841504] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10522.065092] Lustre: Setting parameter fs16456-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs16456-MDT0001 [10522.186852] Lustre: srv-fs16456-MDT0001: No data found on store. Initialize space. [10522.196720] Lustre: Skipped 4 previous similar messages [10522.284772] Lustre: fs16456-MDT0001: new disk, initializing [10522.288142] Lustre: Skipped 3 previous similar messages [10522.419765] Lustre: srv-fs16456-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10522.430271] Lustre: 293996:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16456-MDT0001-mdtlov: can't create new llog: rc = -115 [10522.449224] LustreError: 293996:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16456-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10527.725097] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10535.746233] Lustre: server umount fs16456-MDT0001 complete [10547.691424] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10550.860500] loop1: detected capacity change from 0 to 204800000 [10556.760886] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10557.124798] Lustre: 295565:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16456-MDT0001-mdtlov: can't create new llog: rc = -115 [10557.140504] LustreError: 295565:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16456-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10561.773309] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10566.672992] Lustre: server umount fs16456-MDT0001 complete [10578.405166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10578.426559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10578.428156] Lustre: Skipped 2 previous similar messages [10578.441421] Lustre: Skipped 3 previous similar messages [10582.800038] Lustre: server umount lustre-MDT0000 complete [10592.218829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10592.461223] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10598.405974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10599.835711] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10609.376576] loop1: detected capacity change from 0 to 204800000 [10616.057715] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10616.430714] Lustre: 298014:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16456-MDT0001-mdtlov: can't create new llog: rc = -115 [10616.448184] LustreError: 298014:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16456-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10621.052597] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10625.852581] Lustre: server umount fs16456-MDT0001 complete [10649.062329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10649.068616] LustreError: Skipped 1 previous similar message [10649.076882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10649.080663] Lustre: Skipped 1 previous similar message [10653.710949] Lustre: server umount lustre-OST0000 complete [10664.033916] Lustre: server umount lustre-MDT0000 complete [10667.857132] LustreError: 289790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394442 with bad export cookie 12799082310314031676 [10667.870652] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10667.871197] LustreError: 289790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10668.451682] Lustre: server umount lustre-MDT0001 complete [10682.096844] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [10685.523644] Key type lgssc unregistered [10685.979548] LNet: 300357:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10685.989786] LNetError: 300357:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10686.013053] LNet: Removed LNI 192.168.206.119@tcp [10687.109933] Key type .llcrypt unregistered [10687.117529] Key type ._llcrypt unregistered [10710.472720] Key type ._llcrypt registered [10710.482649] Key type .llcrypt registered [10710.573882] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [10711.866925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10711.899749] alg: No test for adler32 (adler32-zlib) [10713.024782] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [10713.370905] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [10715.073123] Key type lgssc registered [10716.144313] Lustre: Echo OBD driver; http://www.lustre.org/ [10724.142798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10724.172233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10725.535206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10729.588542] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10730.977710] LustreError: 301632:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10731.037657] LustreError: 301632:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10736.107250] LustreError: 301633:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10740.338147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10740.751231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10745.377359] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10753.637877] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10760.018229] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10767.048442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10767.492528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10770.609318] LustreError: 303465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10773.791955] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10776.043384] LustreError: 303464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10781.170495] LustreError: 303463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10782.951363] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10791.287606] LustreError: 303463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10791.312981] LustreError: 303463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10808.252934] loop1: detected capacity change from 0 to 204800000 [10808.508493] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10808.518248] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10808.529532] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10811.774405] LustreError: 303645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10811.791170] LustreError: 303645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [10813.954351] loop1: detected capacity change from 0 to 204800000 [10819.841475] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10819.981177] Lustre: fs7630-OST0000: new disk, initializing [10819.984923] Lustre: srv-fs7630-OST0000: No data found on store. Initialize space. [10819.994243] Lustre: 305368:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10820.075379] Lustre: fs7630-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10827.821540] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10836.207696] Lustre: server umount fs7630-OST0000 complete [10847.601686] LustreError: 303645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10847.624245] LustreError: 303645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 20 previous similar messages [10847.733626] loop1: detected capacity change from 0 to 204800000 [10852.854933] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10853.029573] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10853.265489] Lustre: fs7630-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10859.227684] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10868.208098] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10868.226637] loop2: detected capacity change from 0 to 204800000 [10868.405409] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10868.421377] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10868.438380] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10873.985704] loop2: detected capacity change from 0 to 204800000 [10879.954032] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10880.056718] LustreError: fs7630-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10880.076195] LustreError: fs7630-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10880.105463] LustreError: 308410:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10880.113891] LustreError: 308410:0:(tgt_mount.c:2129:server_put_super()) no obd fs7630-OST0001 [10880.122130] LustreError: 308410:0:(tgt_mount.c:116:server_deregister_mount()) fs7630-OST0001 not registered [10880.166976] Lustre: server umount fs7630-OST0001 complete [10880.174367] LustreError: 308410:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10890.516282] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10890.654327] Lustre: fs7630-OST0001: new disk, initializing [10890.666388] Lustre: srv-fs7630-OST0001: No data found on store. Initialize space. [10890.677956] Lustre: 308960:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10890.767375] Lustre: fs7630-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10897.196935] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10905.265542] Lustre: server umount fs7630-OST0000 complete [10914.172989] LustreError: 303645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10914.193538] LustreError: 303645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [10914.277664] Lustre: server umount fs7630-OST0001 complete [10926.087372] loop1: detected capacity change from 0 to 204800000 [10932.169280] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10932.362392] Lustre: fs7630-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10938.220447] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10944.979733] loop2: detected capacity change from 0 to 204800000 [10951.246229] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10959.840438] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10969.546843] Lustre: server umount fs7630-OST0000 complete [10978.695205] Lustre: server umount fs7630-OST0001 complete [10996.206313] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10996.217449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10996.234274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10996.250472] Lustre: Skipped 1 previous similar message [10996.273746] Lustre: Skipped 1 previous similar message [10997.765902] Lustre: server umount lustre-OST0000 complete [11001.315290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11001.319489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11001.323653] Lustre: Skipped 1 previous similar message [11006.446984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11006.462516] Lustre: Skipped 2 previous similar messages [11007.590726] Lustre: server umount lustre-MDT0000 complete [11010.795165] LustreError: 302376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394785 with bad export cookie 10275510478277333655 [11010.801074] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11010.803437] LustreError: 302376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11011.078661] Lustre: server umount lustre-MDT0001 complete [11021.327272] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [11024.534465] Key type lgssc unregistered [11024.776975] LNet: 314887:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11024.785703] LNetError: 314887:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11024.811786] LNet: Removed LNI 192.168.206.119@tcp [11025.862569] Key type .llcrypt unregistered [11025.865952] Key type ._llcrypt unregistered [11062.509823] Key type ._llcrypt registered [11062.516440] Key type .llcrypt registered [11062.680957] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [11080.166442] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [11081.799375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11081.832601] alg: No test for adler32 (adler32-zlib) [11082.942940] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [11083.207650] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [11084.903519] Key type lgssc registered [11086.393164] Lustre: Echo OBD driver; http://www.lustre.org/ [11129.269554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11129.318509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11130.857395] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11130.935962] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11131.170164] Lustre: lustre-MDT0000: new disk, initializing [11131.374735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11131.445973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11137.616887] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [11148.717079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11148.867635] Lustre: 318748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11148.939981] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11148.948349] Lustre: Skipped 1 previous similar message [11149.100414] Lustre: lustre-MDT0001: new disk, initializing [11149.200933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11149.262702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11149.292708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11153.763176] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [11163.519658] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11169.459933] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11177.062934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11177.406755] Lustre: lustre-OST0000: new disk, initializing [11177.411319] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11177.422290] Lustre: 320714:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11177.528385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11183.908640] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [11186.774526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11186.786512] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11186.944062] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11194.094959] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11196.400536] Lustre: Mounted lustre-client [11197.293782] Lustre: Unmounted lustre-client [11207.648386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11207.659703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11207.670114] LustreError: Skipped 1 previous similar message [11207.689153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11207.706367] Lustre: Skipped 1 previous similar message [11211.755088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11211.772857] Lustre: Skipped 1 previous similar message [11213.827744] Lustre: server umount lustre-OST0000 complete [11220.962616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11220.971420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11221.000290] Lustre: Skipped 1 previous similar message [11221.011398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11221.984321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11224.742702] Lustre: server umount lustre-MDT0000 complete [11228.403920] LustreError: 318740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395003 with bad export cookie 7299607643655099008 [11228.405797] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11228.411850] LustreError: 318740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11228.684398] Lustre: server umount lustre-MDT0001 complete [11240.970119] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [11243.588391] Key type lgssc unregistered [11252.191907] LNet: 1 local NIs in recovery (showing 1): 192.168.206.119@tcp [11253.151480] LNet: 322877:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11253.166972] LNetError: 322877:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11326.960145] LNet: Removed LNI 192.168.206.119@tcp [11327.969408] Key type .llcrypt unregistered [11327.972842] Key type ._llcrypt unregistered [11347.618152] Key type ._llcrypt registered [11347.625841] Key type .llcrypt registered [11347.775815] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [11364.518624] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [11365.912955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11365.983557] alg: No test for adler32 (adler32-zlib) [11367.222809] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [11367.455544] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [11369.175353] Key type lgssc registered [11370.767303] Lustre: Echo OBD driver; http://www.lustre.org/ [11407.298111] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 06:39:39 (1787395179) [11415.468481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11415.516514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11416.884737] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11416.927525] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11417.081718] Lustre: lustre-MDT0000: new disk, initializing [11417.245220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11417.282266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11422.731669] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [11435.622930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11435.785342] Lustre: 326101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11435.860332] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11435.865182] Lustre: Skipped 1 previous similar message [11435.938779] Lustre: lustre-MDT0001: new disk, initializing [11436.086545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11436.145339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11436.175470] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11440.936740] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [11452.527453] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11459.572930] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11466.606965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11466.900576] Lustre: lustre-OST0000: new disk, initializing [11466.908645] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11466.924621] Lustre: 328073:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11467.052228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11472.507509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11472.520171] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11472.702464] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11474.804870] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [11485.184322] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11503.587223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11503.604643] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11503.611310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11503.613794] Lustre: Skipped 1 previous similar message [11508.705432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11508.723481] Lustre: Skipped 2 previous similar messages [11509.229363] Lustre: server umount lustre-OST0000 complete [11517.934942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11517.946949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11517.963880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11518.955406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11520.833666] Lustre: server umount lustre-MDT0000 complete [11525.203429] LustreError: 328072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395299 with bad export cookie 7019927405207269750 [11525.224077] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11525.630374] Lustre: server umount lustre-MDT0001 complete [11539.419201] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [11543.156481] Key type lgssc unregistered [11543.471144] LNet: 329982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11543.481552] LNetError: 329982:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11543.497056] LNet: Removed LNI 192.168.206.119@tcp [11544.506427] Key type .llcrypt unregistered [11544.511810] Key type ._llcrypt unregistered [11567.447236] Key type ._llcrypt registered [11567.449981] Key type .llcrypt registered [11567.598428] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [11569.117035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11569.138430] alg: No test for adler32 (adler32-zlib) [11570.265639] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [11570.540402] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [11572.255343] Key type lgssc registered [11573.721861] Lustre: Echo OBD driver; http://www.lustre.org/ [11582.387312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11582.411932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11583.977686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11589.265429] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [11594.209784] LustreError: 331261:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11594.233037] LustreError: 331261:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11599.341282] LustreError: 331260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11600.975037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11601.567419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11606.366982] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [11615.037412] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11621.277965] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11629.577832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11630.117041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11638.061840] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [11646.760247] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11668.052867] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11669.807554] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11674.101843] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11674.118379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11674.124440] Lustre: Skipped 1 previous similar message [11679.144951] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.19@tcp (stopping) [11679.152330] Lustre: Skipped 2 previous similar messages [11680.685642] Lustre: server umount lustre-MDT0000 complete [11684.218850] LustreError: 331256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11685.287431] LustreError: 331996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395460 with bad export cookie 13972957892715428421 [11685.301985] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11689.322153] LustreError: 331257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11689.338857] LustreError: 331257:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11689.365318] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.19@tcp (stopping) [11689.371491] Lustre: Skipped 3 previous similar messages [11689.442663] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11689.461657] Lustre: Skipped 1 previous similar message [11692.030442] Lustre: server umount lustre-MDT0001 complete [11697.832647] Lustre: server umount lustre-OST0000 complete [11708.592448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11713.669789] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [11716.414577] Lustre: 335465:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.119@tcp [11718.559414] Lustre: server umount MGS complete [11727.162132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11727.704237] LustreError: 336037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11727.739260] LustreError: 336037:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11727.823145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11732.418247] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [11738.082542] LustreError: 336038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11738.108367] LustreError: 336038:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11741.618823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11741.970027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11747.208695] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [11756.880050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11757.284498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11764.438320] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [11775.604428] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11778.735758] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11787.085908] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11789.007454] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11793.384899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11793.387053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11793.407971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11793.415830] Lustre: Skipped 1 previous similar message [11793.419924] Lustre: Skipped 2 previous similar messages [11795.837627] Lustre: server umount lustre-OST0000 complete [11803.626171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11803.647892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11803.654862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11803.661140] Lustre: Skipped 2 previous similar messages [11813.857099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11813.869737] Lustre: Skipped 3 previous similar messages [11814.367447] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11814.639442] Lustre: server umount lustre-MDT0000 complete [11818.261588] LustreError: 337495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395593 with bad export cookie 13972957892715429261 [11818.268099] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11818.283764] LustreError: 337495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11818.640995] Lustre: server umount lustre-MDT0001 complete [11829.877573] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [11832.679609] Key type lgssc unregistered [11832.958817] LNet: 339476:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11832.968532] LNetError: 339476:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11832.991454] LNet: Removed LNI 192.168.206.119@tcp [11833.821538] Key type .llcrypt unregistered [11833.823229] Key type ._llcrypt unregistered [11853.005835] Key type ._llcrypt registered [11853.009869] Key type .llcrypt registered [11853.182393] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [11871.482861] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [11873.597355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11873.631227] alg: No test for adler32 (adler32-zlib) [11874.772176] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [11874.944351] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [11876.631834] Key type lgssc registered [11878.579892] Lustre: Echo OBD driver; http://www.lustre.org/ [11918.793926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11918.861229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11920.163994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11920.209173] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11920.343339] Lustre: lustre-MDT0000: new disk, initializing [11920.426173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11920.463182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11924.781599] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [11935.944522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11936.152569] Lustre: 342639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11936.227833] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11936.234302] Lustre: Skipped 1 previous similar message [11936.321628] Lustre: lustre-MDT0001: new disk, initializing [11936.493632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11936.546363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11936.569697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11941.396449] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [11951.493944] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11958.141899] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11965.791487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11966.139335] Lustre: lustre-OST0000: new disk, initializing [11966.150719] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11966.161648] Lustre: 344606:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11966.265762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11972.833651] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [11973.733460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11973.741539] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11973.964073] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11982.924724] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11987.880893] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11988.059358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11992.856531] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11993.072023] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11999.712665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11999.716037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11999.731572] Lustre: Skipped 1 previous similar message [11999.757776] Lustre: Skipped 1 previous similar message [12001.571663] Lustre: server umount lustre-OST0000 complete [12008.418497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12008.432407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12008.456592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12009.959428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12009.962138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12011.830678] Lustre: server umount lustre-MDT0000 complete [12016.305560] LustreError: 342632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395791 with bad export cookie 9266182452969758551 [12016.309367] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12016.324726] LustreError: 342632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12017.016613] Lustre: server umount lustre-MDT0001 complete [12027.094162] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 06:49:59 (1787395799) [12045.815730] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [12056.261822] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [12093.568082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12093.735693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12093.760552] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12093.867690] Lustre: lustre-MDT0000: new disk, initializing [12093.959538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12093.983869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12097.946859] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [12108.788051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12108.888630] Lustre: 349369:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12108.935618] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12108.940094] Lustre: Skipped 1 previous similar message [12108.995298] Lustre: lustre-MDT0001: new disk, initializing [12109.091528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12109.137686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12109.151574] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12113.455532] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [12124.691727] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12130.909661] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12138.568149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12138.858884] Lustre: lustre-OST0000: new disk, initializing [12138.861865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12138.884271] Lustre: 351335:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12139.032156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12146.760314] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [12148.328779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12148.335856] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12148.486953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12156.172772] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12194.798122] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12194.798694] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12194.833737] LustreError: Skipped 1 previous similar message [12194.840125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12194.847466] Lustre: Skipped 2 previous similar messages [12198.498922] Lustre: server umount lustre-OST0000 complete [12205.027450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12205.034785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12205.039790] Lustre: Skipped 2 previous similar messages [12210.145235] LustreError: 349381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12210.175591] LustreError: 349381:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12211.268659] Lustre: server umount lustre-MDT0000 complete [12215.267546] LustreError: 349377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12215.281988] LustreError: 349377:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12215.874345] LustreError: 349360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395990 with bad export cookie 9266182452969759433 [12215.885472] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12215.893025] LustreError: 349360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12216.249783] Lustre: server umount lustre-MDT0001 complete [12229.795867] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [12232.598348] Key type lgssc unregistered [12232.829493] LNet: 353513:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12232.836918] LNetError: 353513:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12232.874065] LNet: Removed LNI 192.168.206.119@tcp [12233.753084] Key type .llcrypt unregistered [12233.757803] Key type ._llcrypt unregistered [12329.345801] Key type ._llcrypt registered [12329.348200] Key type .llcrypt registered [12329.444847] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [12329.742282] Key type .llcrypt unregistered [12329.744068] Key type ._llcrypt unregistered [12411.228638] Key type ._llcrypt registered [12411.232964] Key type .llcrypt registered [12411.358617] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [12411.603504] Key type .llcrypt unregistered [12411.609198] Key type ._llcrypt unregistered [12443.331797] Key type ._llcrypt registered [12443.333641] Key type .llcrypt registered [12443.401325] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [12443.632329] Key type .llcrypt unregistered [12443.634110] Key type ._llcrypt unregistered [12475.364034] Key type ._llcrypt registered [12475.368367] Key type .llcrypt registered [12475.477332] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [12475.807283] Key type .llcrypt unregistered [12475.809906] Key type ._llcrypt unregistered [12557.523300] Key type ._llcrypt registered [12557.526672] Key type .llcrypt registered [12557.687889] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [12557.956394] Key type .llcrypt unregistered [12557.959177] Key type ._llcrypt unregistered [12589.073950] Key type ._llcrypt registered [12589.076860] Key type .llcrypt registered [12589.221516] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [12589.532223] Key type .llcrypt unregistered [12589.533845] Key type ._llcrypt unregistered [12619.644639] Key type ._llcrypt registered [12619.647524] Key type .llcrypt registered [12619.725468] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [12620.020343] Key type .llcrypt unregistered [12620.023528] Key type ._llcrypt unregistered [12651.375971] Key type ._llcrypt registered [12651.380261] Key type .llcrypt registered [12651.518569] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [12651.897104] Key type .llcrypt unregistered [12651.900837] Key type ._llcrypt unregistered [12683.711269] Key type ._llcrypt registered [12683.718841] Key type .llcrypt registered [12683.848422] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [12684.225384] Key type .llcrypt unregistered [12684.229029] Key type ._llcrypt unregistered [12721.560357] Key type ._llcrypt registered [12721.562968] Key type .llcrypt registered [12721.725671] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [12721.983547] Key type .llcrypt unregistered [12721.986195] Key type ._llcrypt unregistered [12791.327741] Key type ._llcrypt registered [12791.333425] Key type .llcrypt registered [12791.474791] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [12791.757247] Key type .llcrypt unregistered [12791.765197] Key type ._llcrypt unregistered [12821.414114] Key type ._llcrypt registered [12821.416591] Key type .llcrypt registered [12821.522636] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [12821.867618] Key type .llcrypt unregistered [12821.870738] Key type ._llcrypt unregistered [12851.266493] Key type ._llcrypt registered [12851.269866] Key type .llcrypt registered [12851.368631] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [12851.621670] Key type .llcrypt unregistered [12851.623965] Key type ._llcrypt unregistered [12908.245826] Key type ._llcrypt registered [12908.249722] Key type .llcrypt registered [12908.356220] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [12924.580440] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [12925.992757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12926.062077] alg: No test for adler32 (adler32-zlib) [12927.122940] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [12927.349980] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [12929.015608] Key type lgssc registered [12930.640836] Lustre: Echo OBD driver; http://www.lustre.org/ [12968.762582] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12968.807269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12970.028951] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12970.048677] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12970.098899] Lustre: tst250fs-MDT0000: new disk, initializing [12970.155982] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12970.183405] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12974.952488] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [12986.278645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12986.472636] Lustre: 373867:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12986.592921] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12986.596774] Lustre: Skipped 1 previous similar message [12986.693726] Lustre: tst250fs-MDT0001: new disk, initializing [12986.896857] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12986.972725] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12986.994739] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12991.212202] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [12996.992142] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13005.405955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13005.910768] Lustre: tst250fs-OST0000: new disk, initializing [13005.920593] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13005.967594] Lustre: 375501:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13006.152946] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13014.131124] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13014.145085] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13014.325823] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13014.410461] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [13026.395678] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13026.663978] Lustre: tst250fs-OST0001: new disk, initializing [13026.669514] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13026.683101] Lustre: 376373:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13026.842402] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13033.747753] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [13035.675108] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13035.690665] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13035.800541] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13039.809738] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13042.455217] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13042.462802] Lustre: Skipped 2 previous similar messages [13061.604750] Lustre: tst250fs-MDT0000-lwp-OST0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13061.613578] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13061.630672] Lustre: Skipped 1 previous similar message [13061.645402] Lustre: Skipped 3 previous similar messages [13065.971914] Lustre: server umount tst250fs-MDT0000 complete [13069.535800] LustreError: 373859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396844 with bad export cookie 6684986373869376431 [13069.549981] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13069.550622] LustreError: 373859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13069.862363] Lustre: server umount tst250fs-MDT0001 complete [13084.068215] Lustre: server umount tst250fs-OST0000 complete [13098.244690] Lustre: server umount tst250fs-OST0001 complete [13116.579614] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [13124.798252] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [13156.902272] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 07:08:49 (1787396929) [13158.651280] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13176.961294] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [13186.981118] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [13220.968111] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12970 sec ========= 07:09:53 (1787396993) [13222.790595] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:09:55 (1787396995) === [13226.642943] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:09:59 (1787396999) === [13264.008667] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [13266.856339] Key type lgssc unregistered [13267.138507] LNet: 385502:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13267.147681] LNetError: 385502:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13267.164712] LNet: Removed LNI 192.168.206.119@tcp [13268.141153] Key type .llcrypt unregistered [13268.143540] Key type ._llcrypt unregistered