[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 478560512 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2608MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002702] x2apic enabled [ 0.003014] Switched APIC routing to physical x2apic. [ 0.004022] kvm-guest: setup PV IPIs [ 0.008356] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011132] LSM: Security Framework initializing [ 0.013025] Yama: becoming mindful. [ 0.014061] SELinux: Initializing. [ 0.015124] *** VALIDATE selinux *** [ 0.028321] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.035883] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.037180] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.039164] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.040115] *** VALIDATE tmpfs *** [ 0.042533] *** VALIDATE proc *** [ 0.043302] *** VALIDATE cgroup *** [ 0.044016] *** VALIDATE cgroup2 *** [ 0.045354] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.046192] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.047014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.048043] Spectre V2 : User space: Vulnerable [ 0.049013] Speculative Store Bypass: Vulnerable [ 0.052253] debug: unmapping init [mem 0xffffffffb2e59000-0xffffffffb2e60fff] [ 0.056000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.056992] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.058033] ... version: 2 [ 0.059021] ... bit width: 48 [ 0.060014] ... generic registers: 4 [ 0.061018] ... value mask: 0000ffffffffffff [ 0.062021] ... max period: 00007fffffffffff [ 0.063022] ... fixed-purpose events: 3 [ 0.064020] ... event mask: 000000070000000f [ 0.065345] rcu: Hierarchical SRCU implementation. [ 0.067983] smp: Bringing up secondary CPUs ... [ 0.068914] x86: Booting SMP configuration: [ 0.069037] .... node #0, CPUs: #1 #2 #3 [ 0.073249] smp: Brought up 1 node, 4 CPUs [ 0.075017] smpboot: Max logical packages: 1 [ 0.076021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146542] node 0 deferred pages initialised in 67ms [ 0.150318] devtmpfs: initialized [ 0.152281] x86/mm: Memory block size: 128MB [ 0.155929] gcov: version magic: 0x41383552 [ 0.158253] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161078] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.163288] pinctrl core: initialized pinctrl subsystem [ 0.165175] [ 0.165796] ************************************************************* [ 0.168014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170010] ** ** [ 0.172015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.175020] ** ** [ 0.177011] ** This means that this kernel is built to expose internal ** [ 0.179014] ** IOMMU data structures, which may compromise security on ** [ 0.181013] ** your system. ** [ 0.183023] ** ** [ 0.186012] ** If you see this message and you are not debugging the ** [ 0.188011] ** kernel, report this immediately to your vendor! ** [ 0.190019] ** ** [ 0.192013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.194011] ************************************************************* [ 0.196715] NET: Registered protocol family 16 [ 0.199153] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.201059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.204138] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.207087] cpuidle: using governor menu [ 0.209732] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.212553] PCI: Using configuration type 1 for base access [ 0.214130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.223230] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.226024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.230078] cryptd: max_cpu_qlen set to 1000 [ 0.233254] ACPI: Added _OSI(Module Device) [ 0.236024] ACPI: Added _OSI(Processor Device) [ 0.240021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.241015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.246125] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.252617] ACPI: Interpreter enabled [ 0.254076] ACPI: PM: (supports S0 S3 S4 S5) [ 0.256020] ACPI: Using IOAPIC for interrupt routing [ 0.257121] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.261466] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.272690] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.276041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.278018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.282078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.287305] acpiphp: Slot [2] registered [ 0.289110] acpiphp: Slot [5] registered [ 0.291116] acpiphp: Slot [6] registered [ 0.292120] acpiphp: Slot [7] registered [ 0.294125] acpiphp: Slot [8] registered [ 0.295131] acpiphp: Slot [9] registered [ 0.297141] acpiphp: Slot [10] registered [ 0.298118] acpiphp: Slot [3] registered [ 0.300097] acpiphp: Slot [4] registered [ 0.301116] acpiphp: Slot [11] registered [ 0.303090] acpiphp: Slot [12] registered [ 0.304130] acpiphp: Slot [13] registered [ 0.306091] acpiphp: Slot [14] registered [ 0.307100] acpiphp: Slot [15] registered [ 0.309102] acpiphp: Slot [16] registered [ 0.310099] acpiphp: Slot [17] registered [ 0.312104] acpiphp: Slot [18] registered [ 0.313118] acpiphp: Slot [19] registered [ 0.315093] acpiphp: Slot [20] registered [ 0.316101] acpiphp: Slot [21] registered [ 0.317086] acpiphp: Slot [22] registered [ 0.319089] acpiphp: Slot [23] registered [ 0.321077] acpiphp: Slot [24] registered [ 0.322104] acpiphp: Slot [25] registered [ 0.323100] acpiphp: Slot [26] registered [ 0.324073] acpiphp: Slot [27] registered [ 0.325096] acpiphp: Slot [28] registered [ 0.327087] acpiphp: Slot [29] registered [ 0.328086] acpiphp: Slot [30] registered [ 0.329083] acpiphp: Slot [31] registered [ 0.331064] PCI host bridge to bus 0000:00 [ 0.332018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.334020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.335017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.337020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.339019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.341022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.342164] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.344913] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.347000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.354742] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.358953] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.361020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.363018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.366019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.368583] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.370756] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.373039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.375721] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.380014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.392824] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.397016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.402602] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.409000] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.417018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.435023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.448759] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.455016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.461018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.476017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.486841] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.493019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.501018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.513022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.521216] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.533020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.540021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.564074] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.577770] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.583015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.589016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.611017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.619888] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.628023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.636021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.653024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.670216] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.672365] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.675373] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.678381] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.681180] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.685118] iommu: Default domain type: Passthrough [ 0.687560] SCSI subsystem initialized [ 0.689206] ACPI: bus type USB registered [ 0.691158] usbcore: registered new interface driver usbfs [ 0.693129] usbcore: registered new interface driver hub [ 0.695165] usbcore: registered new device driver usb [ 0.697190] pps_core: LinuxPPS API ver. 1 registered [ 0.698015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.702130] PTP clock support registered [ 0.704151] EDAC MC: Ver: 3.0.0 [ 0.706203] PCI: Using ACPI for IRQ routing [ 0.707706] NetLabel: Initializing [ 0.709016] NetLabel: domain hash size = 128 [ 0.710020] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.712126] NetLabel: unlabeled traffic allowed by default [ 0.715221] vgaarb: loaded [ 0.717363] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.719021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.724338] clocksource: Switched to clocksource kvm-clock [ 0.832563] VFS: Disk quotas dquot_6.6.0 [ 0.834533] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.837469] *** VALIDATE ramfs *** [ 0.838954] *** VALIDATE hugetlbfs *** [ 0.840785] pnp: PnP ACPI init [ 0.843446] pnp: PnP ACPI: found 6 devices [ 0.860639] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.864461] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.866838] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.869199] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.871910] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.874577] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.877663] NET: Registered protocol family 2 [ 0.882173] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.888172] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.891969] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.898112] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.901850] TCP: Hash tables configured (established 65536 bind 65536) [ 0.905186] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.910257] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.913873] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.918166] NET: Registered protocol family 1 [ 0.922372] RPC: Registered named UNIX socket transport module. [ 0.924744] RPC: Registered udp transport module. [ 0.926512] RPC: Registered tcp transport module. [ 0.928297] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.930641] NET: Registered protocol family 44 [ 0.932267] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.934469] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.936716] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.939120] PCI: CLS 0 bytes, default 64 [ 0.940840] Unpacking initramfs... [ 2.366779] debug: unmapping init [mem 0xffff9e3ebcc54000-0xffff9e3ebffbffff] [ 2.370895] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.373926] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.376535] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.865043] Initialise system trusted keyrings [ 2.867699] Key type blacklist registered [ 2.871876] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.881544] zbud: loaded [ 2.884726] *** VALIDATE nfs *** [ 2.885858] *** VALIDATE nfs4 *** [ 2.887253] pstore: using deflate compression [ 2.891392] Platform Keyring initialized [ 2.989932] NET: Registered protocol family 38 [ 2.991868] Key type asymmetric registered [ 2.993434] Asymmetric key parser 'x509' registered [ 2.995725] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.999216] io scheduler mq-deadline registered [ 3.000799] io scheduler kyber registered [ 3.002615] io scheduler bfq registered [ 3.004380] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.007790] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.010622] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.013617] ACPI: Power Button [PWRF] [ 3.018852] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.025321] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.038403] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.046603] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.064288] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.092942] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.121986] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.127131] Non-volatile memory driver v1.3 [ 3.129095] Linux agpgart interface v0.103 [ 3.163642] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.166706] vda: detected capacity change from 0 to 76677120 [ 3.180787] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.184391] vdb: detected capacity change from 0 to 1073741824 [ 3.200514] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.204225] vdc: detected capacity change from 0 to 2621440000 [ 3.219397] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.222938] vdd: detected capacity change from 0 to 2621440000 [ 3.241115] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.244326] vde: detected capacity change from 0 to 4294967296 [ 3.259874] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.264414] vdf: detected capacity change from 0 to 4294967296 [ 3.273380] libphy: Fixed MDIO Bus: probed [ 3.278929] usbcore: registered new interface driver usbserial_generic [ 3.281274] usbserial: USB Serial support registered for generic [ 3.283365] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.286513] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.288245] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.290755] mousedev: PS/2 mouse device common for all mice [ 3.293455] rtc_cmos 00:05: RTC can wake from S4 [ 3.296133] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.301612] rtc_cmos 00:05: registered as rtc0 [ 3.304648] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.307107] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.308428] intel_pstate: CPU model not supported [ 3.314254] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.317538] hid: raw HID events driver (C) Jiri Kosina [ 3.320736] usbcore: registered new interface driver usbhid [ 3.322705] usbhid: USB HID core driver [ 3.324318] drop_monitor: Initializing network drop monitor service [ 3.326539] Initializing XFRM netlink socket [ 3.328702] NET: Registered protocol family 10 [ 3.331786] Segment Routing with IPv6 [ 3.333870] NET: Registered protocol family 17 [ 3.336626] mpls_gso: MPLS GSO support [ 3.343698] RAS: Correctable Errors collector initialized. [ 3.346540] AVX version of gcm_enc/dec engaged. [ 3.348700] AES CTR mode by8 optimization enabled [ 3.426304] sched_clock: Marking stable (3426281705, 0)->(4431482876, -1005201171) [ 3.429836] registered taskstats version 1 [ 3.432059] Loading compiled-in X.509 certificates [ 3.434215] zswap: loaded using pool lzo/zbud [ 3.456673] Key type big_key registered [ 3.469350] Key type encrypted registered [ 3.471207] ima: No TPM chip found, activating TPM-bypass! [ 3.473360] ima: Allocated hash algorithm: sha1 [ 3.475077] ima: No architecture policies found [ 3.476690] evm: Initialising EVM extended attributes: [ 3.478958] evm: security.selinux [ 3.480520] evm: security.ima [ 3.481716] evm: security.capability [ 3.482996] evm: HMAC attrs: 0x1 [ 3.485694] rtc_cmos 00:05: setting system clock to 2026-09-05 15:14:33 UTC (1788621273) [ 3.492113] debug: unmapping init [mem 0xffffffffb3e03000-0xffffffffb3ffffff] [ 3.495460] debug: unmapping init [mem 0xffffffffb2b82000-0xffffffffb2e58fff] [ 3.505101] Write protecting the kernel read-only data: 28672k [ 3.508510] debug: unmapping init [mem 0xffffffffb1203000-0xffffffffb13fffff] [ 3.512027] debug: unmapping init [mem 0xffffffffb1b14000-0xffffffffb1bfffff] [ 3.543354] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.555690] systemd[1]: Detected virtualization kvm. [ 3.557523] systemd[1]: Detected architecture x86-64. [ 3.559292] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.588716] systemd[1]: No hostname configured. [ 3.590557] systemd[1]: Set hostname to . [ 3.592673] random: systemd: uninitialized urandom read (16 bytes read) [ 3.595306] systemd[1]: Initializing machine ID from random generator. [ 3.633902] random: ln: uninitialized urandom read (6 bytes read) [ 3.706798] random: systemd: uninitialized urandom read (16 bytes read) [ 3.709025] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.713567] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.719267] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.304294] device-mapper: uevent: version 1.0.3 [ 4.307167] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.003326] random: fast init done [ 5.018093] virtio_net virtio0 ens2: renamed from eth0 [ 5.122460] scsi host0: ata_piix [ 5.149394] scsi host1: ata_piix [ 5.155381] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.159173] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.995967] random: crng init done [ 10.001931] random: 7 urandom warning(s) missed due to ratelimiting [ 11.603594] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 13.041347] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.605806] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.384191] SELinux: Disabled at runtime. [ 16.589189] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.607265] systemd[1]: Detected virtualization kvm. [ 16.609500] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.117801] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.128072] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.158309] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.169545] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.174243] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.195760] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.215903] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ 18.792214] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 19.610562] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.716596] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.734593] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.234908] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.309577] EDAC sbridge: Ver: 1.1.2 [ 23.420072] hrtimer: interrupt took 2241421 ns [ 25.341145] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 25.998251] NFS: Registering the id_resolver key type [ 26.000499] Key type id_resolver registered [ 26.002519] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg608-server login: [ 79.796205] libcfs: loading out-of-tree module taints kernel. [ 79.846442] Key type ._llcrypt registered [ 79.852312] Key type .llcrypt registered [ 79.945846] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 94.942927] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 96.157691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 96.177457] alg: No test for adler32 (adler32-zlib) [ 97.589755] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 98.366977] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 100.128181] Key type lgssc registered [ 101.574053] Lustre: Echo OBD driver; http://www.lustre.org/ [ 115.907125] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 146.557144] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 156.209403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 156.251321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 157.435056] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 157.470555] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 157.536960] Lustre: lustre-MDT0000: new disk, initializing [ 157.593560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 157.606525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 161.242517] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.040298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 172.121973] Lustre: 6500:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 172.165232] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 172.169661] Lustre: Skipped 1 previous similar message [ 172.287989] Lustre: lustre-MDT0001: new disk, initializing [ 172.344692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 172.373353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 172.387070] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 176.169044] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.614786] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 189.818904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 190.154238] Lustre: lustre-OST0000: new disk, initializing [ 190.161223] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 190.165657] Lustre: 8440:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 190.231043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 195.597311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 195.607200] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 195.755602] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 196.613524] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.330101] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.420387] Lustre: lustre-OST0001: new disk, initializing [ 209.423483] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 209.428704] Lustre: 9512:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 209.491688] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 214.606334] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.160756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 217.171875] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 217.233741] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 226.008524] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 235.252087] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 241.166745] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing check_logdir /tmp/testlogs/ [ 245.416476] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing yml_node [ 249.139677] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 251.594464] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 254.018991] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 255.649865] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 11:18:43 EDT 2026 [ 270.680660] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 272.122909] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 288.737134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.745897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 288.757165] Lustre: Skipped 3 previous similar messages [ 288.791160] Lustre: Skipped 3 previous similar messages [ 293.862460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 293.876397] Lustre: Skipped 3 previous similar messages [ 294.062601] Lustre: server umount lustre-MDT0000 complete [ 298.191499] LustreError: 7911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621568 with bad export cookie 17084528378010656985 [ 298.193183] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 298.203370] LustreError: 7911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 298.978151] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 298.993837] Lustre: Skipped 1 previous similar message [ 299.004159] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 299.013427] Lustre: Skipped 1 previous similar message [ 304.097672] LustreError: 12140:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 304.097862] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 304.119326] LustreError: 12140:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 304.140276] Lustre: Skipped 1 previous similar message [ 304.694072] Lustre: server umount lustre-MDT0001 complete [ 309.102512] Lustre: server umount lustre-OST0000 complete [ 312.844256] Lustre: server umount lustre-OST0001 complete [ 319.865358] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 330.103783] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 363.975984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 364.202924] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 364.238594] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 364.318344] Lustre: lustre-MDT0000: new disk, initializing [ 364.412894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 364.431238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 368.212741] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 377.558845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 377.671242] Lustre: 15071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 377.729708] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 377.733572] Lustre: Skipped 1 previous similar message [ 377.827692] Lustre: lustre-MDT0001: new disk, initializing [ 377.925928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 377.951745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 377.963093] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 382.008705] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 389.686082] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 394.747326] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 401.721459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 402.027187] Lustre: lustre-OST0000: new disk, initializing [ 402.032383] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 402.040624] Lustre: 17042:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 402.145424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 404.056937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 404.083394] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 404.260168] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 408.530665] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 417.950803] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 421.795784] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 422.120506] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 426.854560] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 427.174567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 430.049450] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 430.055608] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 430.063877] LustreError: Skipped 1 previous similar message [ 430.084255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 435.680705] Lustre: server umount lustre-OST0000 complete [ 440.289566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.292131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 440.310106] Lustre: Skipped 2 previous similar messages [ 440.319924] Lustre: Skipped 3 previous similar messages [ 446.460934] Lustre: server umount lustre-MDT0000 complete [ 449.840977] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621719 with bad export cookie 17084528378010658518 [ 449.854920] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 449.866331] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 450.309818] Lustre: server umount lustre-MDT0001 complete [ 456.357776] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 11:22:03 (1788621723) [ 475.903126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 476.232436] LustreError: 19915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 476.307826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 481.252702] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 481.768321] LustreError: 19916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.884953] LustreError: 19915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.957825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 495.967793] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 503.105868] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 508.886924] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 510.876551] Lustre: Failing over lustre-MDT0000 [ 511.233352] Lustre: server umount lustre-MDT0000 complete [ 511.970137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 511.972510] LustreError: 19911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 511.984326] Lustre: Skipped 1 previous similar message [ 512.005272] LustreError: 19911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 521.185188] LustreError: 19911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.204261] LustreError: 19911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 527.330493] Lustre: 3638:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788621781/real 1788621781] req@ffff9e3f411a5c00 x1875505439724416/t0(0) o400->MGC192.168.206.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788621797 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 527.376764] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 530.068947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 537.568988] LustreError: 3636:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e3e0192d500 x1875505439727488/t0(0) o250->MGC192.168.206.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 537.606153] LustreError: 20643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 537.631344] LustreError: 20643:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 541.628252] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 543.218124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 543.219403] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 543.254930] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 550.014304] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 557.537845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 557.545320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 557.560034] Lustre: Skipped 1 previous similar message [ 557.581356] Lustre: Skipped 4 previous similar messages [ 559.948095] Lustre: server umount lustre-MDT0000 complete [ 563.400178] LustreError: 19897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621833 with bad export cookie 17084528378010659932 [ 563.413182] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 563.801719] Lustre: server umount lustre-MDT0001 complete [ 571.809482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 572.248427] LustreError: 23251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.270903] LustreError: 23251:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 572.350579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 572.356805] Lustre: Skipped 2 previous similar messages [ 576.723434] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 585.003444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 589.436345] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 596.243505] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 601.703801] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 603.728527] Lustre: Failing over lustre-MDT0000 [ 604.069633] Lustre: server umount lustre-MDT0000 complete [ 606.191250] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 606.210962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 622.270586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 622.414256] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 622.658710] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 626.935886] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 627.701922] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 627.707596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 627.736501] Lustre: Skipped 1 previous similar message [ 627.762473] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 634.891731] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 643.051069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 643.055829] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 643.058178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 643.079142] Lustre: Skipped 2 previous similar messages [ 644.648727] Lustre: server umount lustre-MDT0000 complete [ 648.161262] LustreError: 23247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 648.173451] LustreError: 23247:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 648.256830] LustreError: 23232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621918 with bad export cookie 17084528378010660870 [ 648.260911] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 648.280091] LustreError: 23232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 648.633386] Lustre: server umount lustre-MDT0001 complete [ 656.602512] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 11:25:24 (1788621924) [ 673.533707] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 684.042876] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 717.845280] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 727.075338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 727.335890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 727.358371] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 727.439963] Lustre: lustre-MDT0000: new disk, initializing [ 727.541234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 727.546287] Lustre: Skipped 2 previous similar messages [ 727.558260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 731.682889] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 742.351749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 742.470562] Lustre: 29625:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 742.728453] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 747.065251] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 756.084576] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 761.355068] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 767.267195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 767.460662] Lustre: 31593:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 768.675374] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 768.712277] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 768.758509] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 768.773855] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 772.705965] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 783.549257] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 788.511979] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 804.833301] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 804.834523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 804.843829] LustreError: Skipped 1 previous similar message [ 804.853807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 804.853814] Lustre: Skipped 1 previous similar message [ 810.658985] Lustre: server umount lustre-OST0000 complete [ 814.568780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 821.018424] Lustre: server umount lustre-MDT0000 complete [ 824.409117] LustreError: 29617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622094 with bad export cookie 17084528378010661276 [ 824.418541] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 824.434230] LustreError: 29617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 824.823540] Lustre: server umount lustre-MDT0001 complete [ 834.083960] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 836.845529] Key type lgssc unregistered [ 837.181673] LNet: 33567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.189072] LNetError: 33567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 837.209842] LNet: Removed LNI 192.168.206.108@tcp [ 838.006153] Key type .llcrypt unregistered [ 838.008669] Key type ._llcrypt unregistered [ 860.520941] Key type ._llcrypt registered [ 860.523177] Key type .llcrypt registered [ 860.608511] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 861.722258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 861.765681] alg: No test for adler32 (adler32-zlib) [ 862.852179] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 863.106217] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 864.776959] Key type lgssc registered [ 866.021729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 874.137260] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 874.176518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 875.773259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 879.902405] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 885.221964] LustreError: 34979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 885.243709] LustreError: 34979:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 887.859825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.356546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 892.572709] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 899.618378] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 904.775201] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 911.045701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 911.395633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 913.456492] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 917.406883] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 920.669332] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 924.660896] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 937.932964] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 11:30:05 (1788622205) [ 939.473727] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 941.169882] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 947.838400] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 11:30:15 (1788622215) [ 955.847098] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 11:30:23 (1788622223) [ 956.796207] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 964.653168] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 964.661699] Lustre: Skipped 2 previous similar messages [ 966.427039] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 11:30:33 (1788622233) [ 969.933935] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 969.937807] Lustre: Skipped 3 previous similar messages [ 987.721952] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 987.730158] Lustre: Skipped 6 previous similar messages [ 990.628443] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 11:30:57 (1788622257) [ 993.324441] Lustre: *** cfs_fail_loc=131b, val=2*** [ 996.820340] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1005.609499] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 11:31:12 (1788622272) [ 1007.529851] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1007.536178] Lustre: Skipped 3 previous similar messages [ 1019.894442] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 11:31:27 (1788622287) [ 1036.259083] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1036.269612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1036.292659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1038.305103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1038.307459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1038.309946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1038.330620] Lustre: Skipped 1 previous similar message [ 1039.805896] Lustre: server umount lustre-OST0000 complete [ 1047.015249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1047.024463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1047.033597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1049.837476] Lustre: server umount lustre-MDT0000 complete [ 1053.418323] LustreError: 34959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622323 with bad export cookie 13511181372009440969 [ 1053.423257] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1053.430164] LustreError: 34959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1053.772534] Lustre: server umount lustre-MDT0001 complete [ 1065.794902] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 1068.619822] Key type lgssc unregistered [ 1068.881809] LNet: 42824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1068.900310] LNetError: 42824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1068.927425] LNet: Removed LNI 192.168.206.108@tcp [ 1069.892779] Key type .llcrypt unregistered [ 1069.896216] Key type ._llcrypt unregistered [ 1086.094766] Key type ._llcrypt registered [ 1086.099892] Key type .llcrypt registered [ 1086.214661] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 1101.144758] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1102.374274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1102.438241] alg: No test for adler32 (adler32-zlib) [ 1103.529856] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 1103.745100] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 1105.401112] Key type lgssc registered [ 1106.432480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1136.639417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1136.672622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1137.855541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1137.875788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1137.946578] Lustre: lustre-MDT0000: new disk, initializing [ 1138.004451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1138.038486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1142.073886] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1152.284576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1152.404188] Lustre: 45971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1152.451564] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1152.455537] Lustre: Skipped 1 previous similar message [ 1152.513943] Lustre: lustre-MDT0001: new disk, initializing [ 1152.596182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1152.635742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1152.654323] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1156.902636] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1165.827204] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1171.611679] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1178.091162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1178.286524] Lustre: lustre-OST0000: new disk, initializing [ 1178.290340] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1178.295790] Lustre: 47941:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1178.398985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1184.633273] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1187.429860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1187.453030] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1187.559728] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1193.568548] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1197.988790] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1198.157435] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1202.289754] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1202.511323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1208.302331] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1208.303548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1208.312755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1208.317557] Lustre: Skipped 1 previous similar message [ 1208.336923] LustreError: Skipped 1 previous similar message [ 1210.171825] Lustre: server umount lustre-OST0000 complete [ 1213.922628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1213.935358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1213.943588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1213.951589] Lustre: Skipped 1 previous similar message [ 1218.531905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1218.535074] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1223.649849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1223.657663] Lustre: Skipped 2 previous similar messages [ 1228.256297] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1228.481151] Lustre: server umount lustre-MDT0000 complete [ 1232.069545] LustreError: 45963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622502 with bad export cookie 8598268379223143753 [ 1232.071286] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1232.085266] LustreError: 45963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1232.432674] Lustre: server umount lustre-MDT0001 complete [ 1241.945283] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 11:35:09 (1788622509) [ 1248.126962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1248.660649] LustreError: 50403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1248.669514] LustreError: 50403:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1248.752858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1252.841640] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1253.857189] LustreError: 50404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1258.983506] LustreError: 50403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1260.620637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1260.917378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1264.961668] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1271.525766] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1276.427391] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1282.789418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1283.129611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1288.972544] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1295.964862] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1300.233285] LustreError: 52931:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1301.101726] Lustre: Modifying parameter general.timeout=1 in log params [ 1302.664653] Lustre: Modifying parameter general.timeout=3 in log params [ 1302.668068] Lustre: Skipped 1 previous similar message [ 1304.878203] Lustre: Modifying parameter general.timeout=6 in log params [ 1304.886255] Lustre: Skipped 2 previous similar messages [ 1309.368380] Lustre: Modifying parameter general.timeout=12 in log params [ 1309.373659] Lustre: Skipped 5 previous similar messages [ 1318.066386] Lustre: Modifying parameter general.timeout=23 in log params [ 1318.078343] Lustre: Skipped 10 previous similar messages [ 1334.883860] Lustre: Modifying parameter general.timeout=40 in log params [ 1334.887933] Lustre: Skipped 16 previous similar messages [ 1367.543216] Lustre: Modifying parameter general.timeout=79 in log params [ 1367.548695] Lustre: Skipped 38 previous similar messages [ 1418.470160] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 11:38:05 (1788622685) [ 1421.209258] Lustre: 59864:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1423.254419] Lustre: 59975:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1426.822850] Lustre: 60172:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1426.831103] Lustre: 60172:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1431.501245] Lustre: 60379:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1431.513171] Lustre: 60379:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1433.412204] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule= in log params [ 1433.417382] Lustre: Skipped 58 previous similar messages [ 1457.181171] Lustre: 61736:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1457.200549] LustreError: 61736:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1460.590465] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 11:38:48 (1788622728) [ 1475.046562] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1475.049162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1475.059705] Lustre: Skipped 2 previous similar messages [ 1475.066614] Lustre: Skipped 1 previous similar message [ 1477.574047] Lustre: server umount lustre-MDT0000 complete [ 1480.165726] LustreError: 52709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1480.174176] LustreError: 52709:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1480.543474] LustreError: 51136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622750 with bad export cookie 8598268379223144628 [ 1480.550069] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1480.550482] LustreError: 51136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1480.878616] Lustre: server umount lustre-MDT0001 complete [ 1493.353555] Lustre: server umount lustre-OST0000 complete [ 1516.273497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1516.552235] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1516.820639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1521.260444] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1529.280489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1529.469462] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1529.509254] Lustre: 64267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1534.599359] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1542.027941] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1547.480625] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1553.772234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1553.922840] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1554.154096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1554.157688] Lustre: Skipped 1 previous similar message [ 1559.796850] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1566.203270] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1567.724365] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1567.731232] Lustre: Skipped 15 previous similar messages [ 1567.769308] Lustre: 66828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1574.885097] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1574.895139] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1574.904886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1588.192094] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1588.417326] Lustre: server umount lustre-OST0000 complete [ 1595.873979] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1595.901546] Lustre: Skipped 1 previous similar message [ 1595.907728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1595.921860] Lustre: Skipped 7 previous similar messages [ 1598.684241] Lustre: server umount lustre-MDT0000 complete [ 1600.994474] LustreError: 64277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1601.025593] LustreError: 64277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1602.028637] LustreError: 64257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622872 with bad export cookie 8598268379223147596 [ 1602.032292] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1602.035791] LustreError: 64257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1602.405487] Lustre: server umount lustre-MDT0001 complete [ 1610.454993] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 1612.527773] Key type lgssc unregistered [ 1612.793479] LNet: 68041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1612.799269] LNetError: 68041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1612.814397] LNet: Removed LNI 192.168.206.108@tcp [ 1613.500415] Key type .llcrypt unregistered [ 1613.502365] Key type ._llcrypt unregistered [ 1637.185768] Key type ._llcrypt registered [ 1637.191842] Key type .llcrypt registered [ 1637.303408] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1638.542904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1638.604626] alg: No test for adler32 (adler32-zlib) [ 1639.635785] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 1639.903724] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 1641.624189] Key type lgssc registered [ 1642.866658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1649.834719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1649.854752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1651.252248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1655.174979] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1661.414379] LustreError: 69452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1661.431987] LustreError: 69452:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1662.093603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1662.661522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1666.238980] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1673.924378] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1679.771159] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1686.171947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1686.833312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1692.696176] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1699.290818] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1707.057928] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1716.714157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1716.719842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1716.728049] Lustre: Skipped 2 previous similar messages [ 1716.731263] Lustre: Skipped 1 previous similar message [ 1721.473954] Lustre: server umount lustre-MDT0000 complete [ 1721.825793] LustreError: 71949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1721.858270] LustreError: 71949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1725.040694] LustreError: 72135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622995 with bad export cookie 525766122547094530 [ 1725.052081] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1725.058837] LustreError: 72135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1725.555082] Lustre: server umount lustre-MDT0001 complete [ 1739.774286] Lustre: server umount lustre-OST0000 complete [ 1765.078759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1765.379617] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1765.437855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1765.781063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1770.133763] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1778.082356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1778.163962] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1778.188343] Lustre: 74322:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1778.664117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1783.237881] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1790.715234] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1795.913560] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1802.494419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1802.662268] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1802.954409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1809.109995] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1815.207918] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1816.893280] Lustre: 76897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1818.495980] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1824.752680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1824.754236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1824.771342] LustreError: Skipped 1 previous similar message [ 1824.774704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1824.784616] Lustre: Skipped 1 previous similar message [ 1828.999812] Lustre: server umount lustre-OST0000 complete [ 1834.982199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1834.989327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1835.008296] Lustre: Skipped 2 previous similar messages [ 1835.028870] Lustre: Skipped 2 previous similar messages [ 1839.354721] Lustre: server umount lustre-MDT0000 complete [ 1840.100706] LustreError: 74332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1840.128847] LustreError: 74332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1843.210857] LustreError: 76902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623113 with bad export cookie 525766122547095370 [ 1843.217942] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1843.223033] LustreError: 76902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1843.616154] Lustre: server umount lustre-MDT0001 complete [ 1853.627647] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 1856.496947] Key type lgssc unregistered [ 1856.737362] LNet: 78104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1856.744090] LNetError: 78104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1856.766969] LNet: Removed LNI 192.168.206.108@tcp [ 1857.800901] Key type .llcrypt unregistered [ 1857.803802] Key type ._llcrypt unregistered [ 1883.308205] Key type ._llcrypt registered [ 1883.310018] Key type .llcrypt registered [ 1883.413993] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1884.557437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1884.694104] alg: No test for adler32 (adler32-zlib) [ 1885.770320] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 1886.077401] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 1887.744143] Key type lgssc registered [ 1888.796858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1896.008408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1896.037243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1897.425239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1901.364061] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1907.681456] LustreError: 79515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1907.699270] LustreError: 79515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1908.860517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1909.405389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1913.760512] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1919.645951] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1923.874794] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1929.857992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1930.406932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1936.077802] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1943.674286] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1950.042273] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1950.981710] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1952.428586] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1952.437863] Lustre: Skipped 1 previous similar message [ 1955.028607] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1955.033176] Lustre: Skipped 2 previous similar messages [ 1959.131525] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1959.135440] Lustre: Skipped 4 previous similar messages [ 1967.230073] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1967.234757] Lustre: Skipped 9 previous similar messages [ 1983.360463] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1983.371776] Lustre: Skipped 20 previous similar messages [ 2015.403671] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2015.411119] Lustre: Skipped 43 previous similar messages [ 2033.323750] Lustre: 87606:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2035.829769] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 11:48:23 (1788623303) [ 2037.506094] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2039.213360] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 11:48:26 (1788623306) [ 2042.337698] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2042.345255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2042.357133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2045.411251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2045.412333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2045.433039] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2045.445888] Lustre: Skipped 1 previous similar message [ 2048.422169] Lustre: server umount lustre-OST0000 complete [ 2052.578137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2052.584603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2052.603983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2055.650386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2055.651122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2057.743517] Lustre: server umount lustre-MDT0000 complete [ 2060.770441] LustreError: 81870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2060.782884] LustreError: 81870:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2061.033099] LustreError: 79497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623331 with bad export cookie 11080365324871089642 [ 2061.038271] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2061.042189] LustreError: 79497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2061.365873] Lustre: server umount lustre-MDT0001 complete [ 2070.562955] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 2072.830775] Key type lgssc unregistered [ 2073.079125] LNet: 88854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2073.091417] LNetError: 88854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2073.105413] LNet: Removed LNI 192.168.206.108@tcp [ 2073.833299] Key type .llcrypt unregistered [ 2073.835865] Key type ._llcrypt unregistered [ 2077.992464] Key type ._llcrypt registered [ 2077.995254] Key type .llcrypt registered [ 2078.107593] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2081.564237] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules [ 2082.327026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2082.492825] alg: No test for adler32 (adler32-zlib) [ 2083.617910] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 2083.860793] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 2085.552216] Key type lgssc registered [ 2086.834748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2088.141591] LustreError: 90051:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2103.273413] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2109.819670] LustreError: 90051:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2110.778837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2110.801779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2112.190701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2116.565444] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2122.722746] LustreError: 91075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2122.736826] LustreError: 91075:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2125.479360] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 11:49:52 (1788623392) [ 2127.842697] LustreError: 91076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2129.355187] LustreError: 91107:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2129.366070] LustreError: 91107:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2129.371365] LustreError: 91107:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2129.699973] Lustre: server umount lustre-MDT0000 complete [ 2141.426379] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 2144.083129] Key type lgssc unregistered [ 2144.337087] LNet: 92600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2144.345216] LNetError: 92600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2144.364369] LNet: Removed LNI 192.168.206.108@tcp [ 2145.202539] Key type .llcrypt unregistered [ 2145.206433] Key type ._llcrypt unregistered [ 2159.642410] Key type ._llcrypt registered [ 2159.646611] Key type .llcrypt registered [ 2159.751366] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2160.641068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2160.792132] alg: No test for adler32 (adler32-zlib) [ 2161.888373] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 2162.195635] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 2163.936202] Key type lgssc registered [ 2165.253220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2173.042473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2173.076722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2174.612457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2178.356888] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2184.163482] LustreError: 93792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2184.188888] LustreError: 93792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2185.468070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2186.059260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2189.934625] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2195.443385] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2199.664840] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2204.468294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2204.994155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2209.972084] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2216.155382] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2263.270802] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 11:52:10 (1788623530) [ 2264.925324] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2267.007535] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 11:52:14 (1788623534) [ 2272.739402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2272.744658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.758765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2275.811377] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2275.813965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2275.840090] Lustre: Skipped 2 previous similar messages [ 2277.914278] Lustre: server umount lustre-MDT0000 complete [ 2280.933401] LustreError: 94523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2280.967770] LustreError: 94523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2281.605361] LustreError: 93773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623551 with bad export cookie 15646833896063506726 [ 2281.615535] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2281.620358] LustreError: 93773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2282.211185] Lustre: server umount lustre-MDT0001 complete [ 2296.653854] Lustre: server umount lustre-OST0000 complete [ 2308.612433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2308.992227] LustreError: 98067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2309.236421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2313.744272] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2314.721519] LustreError: 98068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2314.742189] LustreError: 98068:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2319.849726] LustreError: 98067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2321.783591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2322.629079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2326.863932] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2333.573047] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2339.525776] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2352.494225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2352.689098] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2352.696836] LustreError: 98057:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2352.734893] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2352.745445] LustreError: 100150:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2352.757144] LustreError: 100150:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2352.763146] LustreError: 100150:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2352.820837] Lustre: server umount lustre-OST0000 complete [ 2352.825651] LustreError: 100150:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2361.118259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2361.303421] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2361.312740] LustreError: 98055:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2361.341048] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2361.347491] LustreError: 100615:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2361.351958] LustreError: 100615:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2361.355658] LustreError: 100615:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2361.415380] Lustre: server umount lustre-OST0000 complete [ 2361.417526] LustreError: 100615:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2370.096208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2370.267577] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2370.274475] Lustre: Found index 0 for lustre-OST0000, updating log [ 2370.281343] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2370.387280] Lustre: lustre-OST0000: new disk, initializing [ 2370.391558] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2370.398244] Lustre: 101150:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2370.763368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2377.171661] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2380.602326] Lustre: Failing over lustre-OST0000 [ 2380.779297] Lustre: server umount lustre-OST0000 complete [ 2385.379516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2385.393343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2385.398448] Lustre: Skipped 2 previous similar messages [ 2385.418292] Lustre: Skipped 1 previous similar message [ 2390.499242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2390.507612] Lustre: Skipped 1 previous similar message [ 2394.679894] Lustre: server umount lustre-MDT0000 complete [ 2397.681167] LustreError: 98063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2397.705317] LustreError: 98063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2398.229662] LustreError: 98047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623668 with bad export cookie 15646833896063509470 [ 2398.241816] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2398.251978] LustreError: 98047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2398.687585] Lustre: server umount lustre-MDT0001 complete [ 2407.089750] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 11:54:34 (1788623674) [ 2423.400174] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2432.823039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2433.158593] LustreError: 103496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2433.251461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2437.524144] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2445.166446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2450.632418] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2453.857660] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2460.292951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2461.077412] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2467.577178] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2470.312299] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2470.326355] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2470.352428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2475.491840] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:55 [ 2475.510191] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2475.738957] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2475.795415] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2475.859937] Lustre: lustre-OST0001: new disk, initializing [ 2475.869791] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2475.888681] Lustre: 105815:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2476.108085] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2476.114245] Lustre: Skipped 2 previous similar messages [ 2478.194659] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:02 [ 2478.226757] Lustre: Skipped 1 previous similar message [ 2478.233080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2478.240967] LustreError: Skipped 1 previous similar message [ 2478.250855] Lustre: lustre-OST0000: Recovery over after 0:08, of 1 clients 1 recovered and 0 were evicted. [ 2478.258618] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2478.274332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2482.895115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2482.906949] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2483.070334] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2484.181731] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2492.176756] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2498.839744] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2503.663151] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2503.676292] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2503.684910] Lustre: Skipped 1 previous similar message [ 2503.708778] Lustre: Skipped 4 previous similar messages [ 2509.104600] Lustre: server umount lustre-MDT0001 complete [ 2512.796314] LustreError: 103492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.825823] LustreError: 103492:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2518.484987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2518.648847] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2518.663769] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2518.734192] Lustre: 103483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2520.022394] LustreError: 107341:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2520.048656] LustreError: 107341:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2520.063257] LustreError: 103488:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2520.258661] LustreError: 107355:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.108@tcp: cfg command failed: rc = -17 [ 2520.277429] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2520.277429] [ 2524.618413] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2524.657182] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2535.930756] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 11:56:43 (1788623803) [ 2553.282145] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2571.312669] Lustre: 109427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2594.107868] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2597.968951] Lustre: 110563:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2677.732303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2677.740762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2677.758019] Lustre: Skipped 1 previous similar message [ 2677.771982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2677.780642] Lustre: Skipped 4 previous similar messages [ 2682.485320] Lustre: server umount lustre-MDT0000 complete [ 2683.363684] LustreError: 103491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2683.398233] LustreError: 103491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2685.952709] LustreError: 103477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623955 with bad export cookie 15646833896063510471 [ 2685.957411] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2685.970882] LustreError: 103477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2686.299256] Lustre: server umount lustre-MDT0001 complete [ 2700.634073] Lustre: server umount lustre-OST0000 complete [ 2713.364899] Lustre: server umount lustre-OST0001 complete [ 2718.288184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2727.392931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2735.585892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2744.300200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2763.220838] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2773.221531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2773.242801] Lustre: lustre-MDT0000: reset Object Index mappings [ 2773.689667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2773.693839] Lustre: Skipped 1 previous similar message [ 2778.328562] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2785.864430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2785.879103] Lustre: lustre-MDT0001: reset Object Index mappings [ 2790.279220] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2793.165501] Lustre: 115673:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2799.074920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2805.417226] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2811.878632] LustreError: 116029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2811.891012] LustreError: 116029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2811.901040] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 2813.528763] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2818.119346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 2818.124474] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 2818.231331] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2819.782524] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2826.819073] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2830.080552] Lustre: 117549:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2874.689915] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 12:02:22 (1788624142) [ 2904.033610] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2904.042515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2904.063698] Lustre: Skipped 3 previous similar messages [ 2904.072267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2904.080870] Lustre: Skipped 4 previous similar messages [ 2915.296140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2915.541352] Lustre: server umount lustre-MDT0000 complete [ 2919.227034] LustreError: 114494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624189 with bad export cookie 15646833896063557546 [ 2919.233134] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2919.238271] LustreError: 114494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2920.423038] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2920.445468] Lustre: Skipped 3 previous similar messages [ 2925.721303] Lustre: server umount lustre-MDT0001 complete [ 2935.625117] Lustre: server umount lustre-OST0000 complete [ 2953.184111] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2953.328913] Lustre: server umount lustre-OST0001 complete [ 2958.751508] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 2967.899858] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 3002.336088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3002.548665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3002.576780] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3002.658091] Lustre: lustre-MDT0000: new disk, initializing [ 3002.735047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3002.743319] Lustre: Skipped 3 previous similar messages [ 3002.756675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3007.216450] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3013.775527] Lustre: server umount lustre-MDT0000 complete [ 3022.238800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3022.450913] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3022.464701] Lustre: 121820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3022.478254] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3026.947982] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3031.943595] Lustre: server umount lustre-MDT0000 complete [ 3039.356126] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 12:05:07 (1788624307) [ 3040.825621] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3042.331417] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 12:05:10 (1788624310) [ 3059.121091] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 3069.185826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3073.720210] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3081.920996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3082.007542] Lustre: 123869:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3082.048673] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3082.050899] Lustre: Skipped 1 previous similar message [ 3082.094389] Lustre: lustre-MDT0001: new disk, initializing [ 3082.173930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3082.184594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3086.153988] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3090.759653] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3096.798628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3097.200363] Lustre: lustre-OST0000: new disk, initializing [ 3097.204761] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3097.212552] Lustre: 125449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3098.913971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3098.926484] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3099.094614] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3103.865493] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3114.700624] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3114.901688] Lustre: lustre-OST0001: new disk, initializing [ 3114.906530] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3114.913808] Lustre: 126319:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3116.002098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3116.011214] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3116.161703] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3121.587729] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3130.917229] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3134.420667] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3455.432707] Lustre: 132721:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3455.442238] Lustre: 132721:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3455.447807] Lustre: 132721:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3455.456989] Lustre: 132721:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3455.463051] Lustre: 132721:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3455.471684] Lustre: 132721:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3720.323530] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 12:16:26 (1788624986) [ 3735.524082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3735.527424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3735.543630] Lustre: Skipped 4 previous similar messages [ 3735.554361] Lustre: Skipped 18 previous similar messages [ 3740.643090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3740.654283] Lustre: Skipped 4 previous similar messages [ 3741.662303] Lustre: server umount lustre-MDT0000 complete [ 3745.763201] LustreError: 125440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3745.812824] LustreError: 125440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3746.035846] LustreError: 123861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625016 with bad export cookie 15646833896063623864 [ 3746.051253] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3746.067681] LustreError: 123861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3750.883678] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3750.907549] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3750.913483] Lustre: Skipped 1 previous similar message [ 3753.028327] Lustre: server umount lustre-MDT0001 complete [ 3771.360112] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3771.563035] Lustre: server umount lustre-OST0000 complete [ 3781.390042] Lustre: server umount lustre-OST0001 complete [ 3787.319486] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 3796.035653] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 3829.243174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3829.435874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3829.461955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3829.520063] Lustre: lustre-MDT0000: new disk, initializing [ 3829.579304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3829.590214] Lustre: Skipped 5 previous similar messages [ 3829.612457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3833.961920] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3842.812577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3842.934165] Lustre: 135779:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3842.987474] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3842.991821] Lustre: Skipped 1 previous similar message [ 3843.065101] Lustre: lustre-MDT0001: new disk, initializing [ 3843.184042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3843.197231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3847.156154] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3855.922414] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3861.272874] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3867.603275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3867.871450] Lustre: lustre-OST0000: new disk, initializing [ 3867.876711] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3867.884930] Lustre: 137746:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3869.472967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3869.482769] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3869.572790] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3874.528292] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3883.945610] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3893.295518] Lustre: lustre-MDD0000: changelog on [ 3895.787768] Lustre: lustre-MDD0001: changelog on [ 5448.888855] Lustre: lustre-MDD0001: changelog off [ 5450.529975] Lustre: 136961:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5451.240828] Lustre: lustre-MDD0000: changelog off [ 5456.866030] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5456.871642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5456.875977] LustreError: Skipped 1 previous similar message [ 5456.877647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5456.884123] Lustre: Skipped 2 previous similar messages [ 5456.900098] Lustre: Skipped 3 previous similar messages [ 5461.987149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5467.106186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5467.117648] Lustre: Skipped 2 previous similar messages [ 5471.200958] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5471.505654] Lustre: server umount lustre-OST0000 complete [ 5476.321378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5476.331122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5476.358357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5489.120369] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5489.379282] Lustre: server umount lustre-MDT0000 complete [ 5492.706761] LustreError: 136961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5492.730655] LustreError: 136961:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5493.696312] LustreError: 135772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626763 with bad export cookie 15646833896064437614 [ 5493.702640] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5493.729074] LustreError: 135772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5494.005715] Lustre: server umount lustre-MDT0001 complete [ 5507.831940] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 5510.887638] Key type lgssc unregistered [ 5511.217656] LNet: 146827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5511.226498] LNetError: 146827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5511.258043] LNet: Removed LNI 192.168.206.108@tcp [ 5512.163053] Key type .llcrypt unregistered [ 5512.164642] Key type ._llcrypt unregistered [ 5533.518568] Key type ._llcrypt registered [ 5533.522478] Key type .llcrypt registered [ 5533.614463] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 5547.873353] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5549.097128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5549.128234] alg: No test for adler32 (adler32-zlib) [ 5550.238479] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 5550.441189] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 5552.072125] Key type lgssc registered [ 5553.008685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5583.438206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5583.482155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5584.724735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5584.778274] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5584.938336] Lustre: lustre-MDT0000: new disk, initializing [ 5585.040124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5585.086956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5589.664151] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5598.948930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5599.082336] Lustre: 150026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5599.133140] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5599.138031] Lustre: Skipped 1 previous similar message [ 5599.236465] Lustre: lustre-MDT0001: new disk, initializing [ 5599.363911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5599.409151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5599.425504] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5603.089344] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5610.653630] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5615.992654] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5622.192986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5622.558872] Lustre: lustre-OST0000: new disk, initializing [ 5622.562779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5622.568594] Lustre: 151996:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5622.683174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5628.907341] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5632.595935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5632.623185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5632.767151] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5638.583303] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5643.756873] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_module obdecho/obdecho [ 5645.311425] LustreError: 152998:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5645.319979] LustreError: 152998:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5646.953385] LustreError: 153094:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5658.594494] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5658.596988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5658.610677] Lustre: Skipped 2 previous similar messages [ 5658.622848] Lustre: Skipped 2 previous similar messages [ 5663.715426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5663.720387] Lustre: Skipped 2 previous similar messages [ 5663.894303] Lustre: server umount lustre-MDT0000 complete [ 5667.053911] LustreError: 151995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626937 with bad export cookie 3492726633357380049 [ 5667.062513] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5667.070133] LustreError: 151995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5667.374860] Lustre: server umount lustre-MDT0001 complete [ 5681.177958] Lustre: server umount lustre-OST0000 complete [ 5699.544698] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 5709.324899] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5742.700943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5742.881565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5742.909360] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5742.970912] Lustre: lustre-MDT0000: new disk, initializing [ 5743.049287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5743.065979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5747.593172] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5757.389677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5757.530898] Lustre: 156808:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5757.608538] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5757.611833] Lustre: Skipped 1 previous similar message [ 5757.672581] Lustre: lustre-MDT0001: new disk, initializing [ 5757.778225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5757.816303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5757.830631] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5761.753285] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5770.045520] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5774.425760] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5779.654905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5779.822466] Lustre: lustre-OST0000: new disk, initializing [ 5779.825767] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5779.831355] Lustre: 158775:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5779.911392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5784.884370] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5789.751031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5789.778867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5789.878497] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5793.412253] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5797.164611] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5797.324793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5800.763066] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5800.964892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5805.540827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5805.541301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5805.556685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5805.559373] Lustre: Skipped 1 previous similar message [ 5808.477613] Lustre: server umount lustre-OST0000 complete [ 5814.241070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5814.246039] LustreError: Skipped 1 previous similar message [ 5814.248968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5814.258413] Lustre: Skipped 1 previous similar message [ 5814.263771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5817.602522] Lustre: server umount lustre-MDT0000 complete [ 5820.418207] LustreError: 156801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627090 with bad export cookie 3492726633357381008 [ 5820.424440] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5820.433822] LustreError: 156801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5820.745655] Lustre: server umount lustre-MDT0001 complete [ 5823.593859] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 12:51:31 (1788627091) [ 5824.741258] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5826.122933] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 12:51:33 (1788627093) [ 5832.168317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5832.700755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5836.791438] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5837.818679] LustreError: 161186:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5837.859869] LustreError: 161186:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5842.915851] LustreError: 161185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5843.939753] LustreError: 161186:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5845.143861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5849.883674] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5856.139379] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5861.272953] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5866.857277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5867.231968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5867.235360] Lustre: Skipped 1 previous similar message [ 5872.292805] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5878.688588] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5883.883569] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5883.894307] Lustre: Skipped 2 previous similar messages [ 5883.898031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5883.904656] Lustre: Skipped 4 previous similar messages [ 5886.269883] Lustre: server umount lustre-MDT0000 complete [ 5889.001691] LustreError: 161185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5889.030971] LustreError: 161185:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5889.747634] LustreError: 161167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627159 with bad export cookie 3492726633357381883 [ 5889.756079] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5889.763362] LustreError: 161167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5894.113347] LustreError: 163025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5894.114655] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5894.135553] LustreError: 163025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5894.160187] Lustre: Skipped 1 previous similar message [ 5894.172869] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5894.175681] Lustre: Skipped 1 previous similar message [ 5904.352133] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5904.596478] Lustre: server umount lustre-MDT0001 complete [ 5910.275239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5921.328297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5921.589354] LustreError: 165028:0:(llog.c:1655:llog_backup()) MGC192.168.206.108@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5921.597990] Lustre: 165028:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.108@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5924.233389] LustreError: 165033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5924.262452] LustreError: 165033:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5940.612903] LustreError: 165050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5940.634111] LustreError: 165050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 5975.430511] LustreError: 165034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5975.445754] LustreError: 165034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 24 previous similar messages [ 5982.176566] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5982.181551] Lustre: 165028:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.108@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6023.169725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6027.624902] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6028.262665] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6035.351263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6040.546524] LustreError: 165033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6040.563709] LustreError: 165033:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 48 previous similar messages [ 6045.667773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6064.521787] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6065.646549] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6072.321957] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6073.956036] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6079.304387] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6081.032585] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6101.481330] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6101.498798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6101.504167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6101.513459] Lustre: Skipped 6 previous similar messages [ 6103.469089] Lustre: server umount lustre-MDT0000 complete [ 6107.147541] LustreError: 161165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627377 with bad export cookie 3492726633357388379 [ 6107.159589] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6107.160377] LustreError: 161165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6107.825730] Lustre: server umount lustre-MDT0001 complete [ 6121.448855] Lustre: server umount lustre-OST0000 complete [ 6142.533638] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 6150.427184] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6179.619538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6179.824389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6179.845773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6179.940867] Lustre: lustre-MDT0000: new disk, initializing [ 6180.023098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6180.028828] Lustre: Skipped 1 previous similar message [ 6180.040829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6183.918949] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6194.222568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6194.357491] Lustre: 170136:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6194.553903] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6198.582648] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6207.138128] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6212.498452] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6219.304554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6219.672509] Lustre: lustre-OST0000: new disk, initializing [ 6219.674849] Lustre: Skipped 1 previous similar message [ 6219.677629] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6219.682093] Lustre: Skipped 2 previous similar messages [ 6219.686972] Lustre: 172106:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6225.998595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6226.020126] Lustre: Skipped 1 previous similar message [ 6226.034793] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6226.209575] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6227.304398] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6237.196887] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6241.192031] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6241.421552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6245.772456] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6246.065697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6252.003019] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6252.004341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6252.009942] LustreError: Skipped 1 previous similar message [ 6252.011945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6252.040936] Lustre: Skipped 3 previous similar messages [ 6262.241574] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6262.464694] Lustre: server umount lustre-OST0000 complete [ 6272.708722] Lustre: server umount lustre-MDT0000 complete [ 6276.610744] LustreError: 170127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627546 with bad export cookie 3492726633357405186 [ 6276.621841] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6276.626514] LustreError: 170127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6277.059388] Lustre: server umount lustre-MDT0001 complete [ 6285.892297] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 12:59:13 (1788627553) [ 6291.851639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6292.534378] LustreError: 174517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6292.552079] LustreError: 174517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 6296.529249] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6305.243462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6311.220514] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6321.543947] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6326.708230] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6332.372433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6338.031682] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6345.877407] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6359.504465] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 13:00:26 (1788627626) [ 6366.689402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6366.703852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6366.708296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6366.734893] Lustre: Skipped 3 previous similar messages [ 6366.755672] Lustre: Skipped 11 previous similar messages [ 6370.250926] Lustre: server umount lustre-OST0000 complete [ 6388.192133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6388.454533] Lustre: server umount lustre-MDT0000 complete [ 6392.004521] LustreError: 174498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627662 with bad export cookie 3492726633357406068 [ 6392.011831] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6392.014768] LustreError: 174498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6392.429189] Lustre: server umount lustre-MDT0001 complete [ 6402.542580] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 6405.049532] Key type lgssc unregistered [ 6405.313765] LNet: 178257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6405.335348] LNetError: 178257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6405.356799] LNet: Removed LNI 192.168.206.108@tcp [ 6406.072720] Key type .llcrypt unregistered [ 6406.074810] Key type ._llcrypt unregistered [ 6423.297185] Key type ._llcrypt registered [ 6423.299517] Key type .llcrypt registered [ 6423.399619] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6424.239607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6424.269116] alg: No test for adler32 (adler32-zlib) [ 6425.353940] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 6425.682437] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 6427.352172] Key type lgssc registered [ 6428.313395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6435.368913] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6435.384437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6450.912390] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6456.032414] LustreError: 179506:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.108@tcp: failed processing log, type 4: rc = -110 [ 6461.155371] LustreError: 179506:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6461.179401] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6461.184576] LustreError: MGC192.168.206.108@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6461.198595] LustreError: 179506:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6461.215046] LustreError: 179506:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 6461.227469] LustreError: 179506:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 6461.327857] Lustre: server umount lustre-OST0000 complete [ 6461.335716] LustreError: 179506:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6469.106240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6470.450557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6474.449684] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6480.868285] LustreError: 180006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6480.885445] LustreError: 180006:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6482.304600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6482.796644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6487.161271] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6494.164303] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6499.594598] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6506.662741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6506.875447] LustreError: 181822:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6506.883753] Lustre: 181822:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.108@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6507.042285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6513.593631] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6521.495506] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6523.872987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6523.879672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6523.887863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6525.410631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6525.413676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6525.419201] Lustre: Skipped 1 previous similar message [ 6525.438864] Lustre: Skipped 1 previous similar message [ 6529.816440] Lustre: server umount lustre-MDT0000 complete [ 6530.535469] LustreError: 180007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6530.549152] LustreError: 180007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6532.586164] LustreError: 179988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627802 with bad export cookie 5127686301869145424 [ 6532.592471] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6532.595624] LustreError: 179988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6532.850300] Lustre: server umount lustre-MDT0001 complete [ 6545.156873] Lustre: server umount lustre-OST0000 complete [ 6554.669357] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 6564.830155] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6594.332425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6594.594979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6594.633362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6594.729421] Lustre: lustre-MDT0000: new disk, initializing [ 6594.784805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6594.798566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6598.795279] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6607.484416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6607.588630] Lustre: 185347:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6607.633349] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6607.636402] Lustre: Skipped 1 previous similar message [ 6607.694711] Lustre: lustre-MDT0001: new disk, initializing [ 6607.789971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6607.817413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6607.826402] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6611.278324] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6619.256275] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6623.886460] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6629.287993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6629.477534] Lustre: lustre-OST0000: new disk, initializing [ 6629.481989] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6629.488701] Lustre: 187314:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6629.565721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6636.123949] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6636.716860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6636.740529] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6636.919308] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6646.579357] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6650.810344] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6651.089593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6654.773383] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6654.931487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6657.505794] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6657.508966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6657.513393] LustreError: Skipped 1 previous similar message [ 6657.515203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6657.531109] Lustre: Skipped 1 previous similar message [ 6657.535097] Lustre: Skipped 1 previous similar message [ 6662.627601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6662.639564] Lustre: Skipped 2 previous similar messages [ 6662.814783] Lustre: server umount lustre-OST0000 complete [ 6667.745694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6667.761281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6672.509652] Lustre: server umount lustre-MDT0000 complete [ 6672.872618] LustreError: 185357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6672.898572] LustreError: 185357:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6675.975762] LustreError: 187313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627945 with bad export cookie 5127686301869146145 [ 6675.980472] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6675.992990] LustreError: 187313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6676.447892] Lustre: server umount lustre-MDT0001 complete [ 6684.429202] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 13:05:52 (1788627952) [ 6700.923504] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 6703.443245] Key type lgssc unregistered [ 6703.783165] LNet: 190257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6703.787958] LNetError: 190257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6704.815087] LNet: Removed LNI 192.168.206.108@tcp [ 6705.625707] Key type .llcrypt unregistered [ 6705.631270] Key type ._llcrypt unregistered [ 6723.374731] Key type ._llcrypt registered [ 6723.377017] Key type .llcrypt registered [ 6723.517552] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6724.449294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6724.584429] alg: No test for adler32 (adler32-zlib) [ 6725.640432] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 6725.887765] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 6727.544272] Key type lgssc registered [ 6728.573620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6734.788420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6734.816769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6736.216312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6739.971225] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6746.596767] LustreError: 191471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6746.612937] LustreError: 191471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6747.376864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6747.625486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6751.109449] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6757.967408] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6763.397931] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6771.233200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6771.470680] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6771.473496] LustreError: 193335:0:(llog.c:1696:llog_backup()) MGC192.168.206.108@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6771.479266] Lustre: 193335:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.108@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6771.640572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6778.529328] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6785.390668] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6788.579172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6788.584485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6788.610069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6792.162041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6792.175981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6792.181783] Lustre: Skipped 1 previous similar message [ 6792.188903] Lustre: Skipped 2 previous similar messages [ 6797.283306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6797.293340] Lustre: Skipped 2 previous similar messages [ 6801.377503] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6801.524799] Lustre: server umount lustre-MDT0000 complete [ 6802.402812] LustreError: 191466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6802.433046] LustreError: 191466:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6805.074031] LustreError: 192204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628075 with bad export cookie 2186301178315295213 [ 6805.082884] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6805.088726] LustreError: 192204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6805.398522] Lustre: server umount lustre-MDT0001 complete [ 6818.320604] Lustre: server umount lustre-OST0000 complete [ 6827.720411] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 6838.841582] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6874.034340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6874.330653] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6874.373672] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6874.513610] Lustre: lustre-MDT0000: new disk, initializing [ 6874.677369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6874.707564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6879.513300] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6889.248517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6889.391288] Lustre: 196860:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6889.438207] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6889.439619] Lustre: Skipped 1 previous similar message [ 6889.443367] LustreError: 197644:0:(llog.c:1696:llog_backup()) MGC192.168.206.108@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 6889.448379] LustreError: 197644:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 6889.451333] Lustre: 197644:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.108@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 6889.470702] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6889.474073] Lustre: Skipped 1 previous similar message [ 6889.563258] Lustre: lustre-MDT0001: new disk, initializing [ 6889.657503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6889.695667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6889.709705] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6894.222206] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6903.342499] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6909.033194] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6915.341063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6915.606700] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6915.608518] Lustre: Skipped 1 previous similar message [ 6915.612325] LustreError: 198805:0:(llog.c:1696:llog_backup()) MGC192.168.206.108@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6915.617197] LustreError: 198805:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 6915.621953] Lustre: 198805:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.108@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 6915.628945] Lustre: 198805:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6915.646853] Lustre: lustre-OST0000: new disk, initializing [ 6915.650067] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6915.653880] Lustre: 198828:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6915.749024] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6922.308656] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6922.829764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6922.841352] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6922.975349] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6931.491117] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6936.126659] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6936.452498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6941.149370] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6941.359814] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6943.719053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6943.725298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6943.732199] Lustre: Skipped 1 previous similar message [ 6943.732518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6943.752721] Lustre: Skipped 1 previous similar message [ 6948.840812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6948.848846] Lustre: Skipped 1 previous similar message [ 6949.501810] Lustre: server umount lustre-OST0000 complete [ 6953.953802] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6958.709763] Lustre: server umount lustre-MDT0000 complete [ 6959.073789] LustreError: 196872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6959.094980] LustreError: 196872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6962.256698] LustreError: 196851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628232 with bad export cookie 2186301178315295934 [ 6962.264370] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6962.274923] LustreError: 196851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6962.685285] Lustre: server umount lustre-MDT0001 complete [ 6970.578768] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 13:10:38 (1788628238) [ 6984.781535] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6994.633380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6995.028091] LustreError: 201953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6995.059452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6999.299639] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7000.563068] LustreError: 201954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7007.509073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7012.087681] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7014.593244] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7020.258238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7026.706697] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7034.473354] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7034.651569] Lustre: lustre-OST0001: new disk, initializing [ 7034.654563] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7034.661176] Lustre: 204260:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7034.761349] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7034.769361] Lustre: Skipped 2 previous similar messages [ 7037.756713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7037.763113] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7037.824828] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7040.353319] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7049.931483] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7053.992393] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7055.777758] Lustre: DEBUG MARKER: ADD OST3 [ 7060.321178] loop: module loaded [ 7060.379320] loop0: detected capacity change from 0 to 204800000 [ 7060.529514] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7060.537716] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7060.555042] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7066.715198] loop0: detected capacity change from 0 to 204800000 [ 7067.215390] Lustre: DEBUG MARKER: STOP OST3 [ 7071.999841] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7072.231479] Lustre: lustre-OST0003: new disk, initializing [ 7072.238897] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7072.247764] Lustre: 206263:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7073.512457] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7073.612628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7073.620754] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7073.655325] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7075.553945] LustreError: 206409:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7075.919726] Lustre: Failing over lustre-OST0003 [ 7076.031219] Lustre: server umount lustre-OST0003 complete [ 7076.427621] LustreError: 204446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7076.450909] LustreError: 204446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7078.760784] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7078.893079] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7078.901211] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7078.902176] Lustre: Skipped 1 previous similar message [ 7086.263740] Lustre: DEBUG MARKER: START OST3 again [ 7092.180841] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7092.413962] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7093.679631] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7094.194977] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7094.198966] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7097.589723] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7119.618783] Lustre: Failing over lustre-OST0003 [ 7119.801780] Lustre: server umount lustre-OST0003 complete [ 7119.845936] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7119.869077] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7119.878872] LustreError: 203440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7119.898374] Lustre: Skipped 2 previous similar messages [ 7119.934454] LustreError: 203440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 7138.790104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7138.794908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7138.808359] Lustre: Skipped 2 previous similar messages [ 7138.823800] Lustre: Skipped 5 previous similar messages [ 7143.179359] Lustre: server umount lustre-MDT0000 complete [ 7146.475884] LustreError: 201934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628416 with bad export cookie 2186301178315296816 [ 7146.481791] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7146.487213] LustreError: 201934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7152.879409] Lustre: server umount lustre-MDT0001 complete [ 7155.897980] Lustre: server umount lustre-OST0000 complete [ 7158.955352] Lustre: server umount lustre-OST0001 complete [ 7164.851286] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 7173.282818] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 7210.121131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7210.349702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7210.367854] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7210.455398] Lustre: lustre-MDT0000: new disk, initializing [ 7210.534787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7210.544085] Lustre: Skipped 2 previous similar messages [ 7210.562949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7214.838922] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7224.105475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7224.191587] Lustre: 211722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7224.416303] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7228.801915] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7237.029569] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7242.530721] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7248.919804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7249.234468] Lustre: lustre-OST0000: new disk, initializing [ 7249.238391] Lustre: Skipped 1 previous similar message [ 7249.242299] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7249.246901] Lustre: Skipped 2 previous similar messages [ 7249.262385] Lustre: 213690:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7250.705149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7250.719364] Lustre: Skipped 1 previous similar message [ 7250.723566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7250.835664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7255.208789] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7263.377351] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7267.479584] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7267.671478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7271.916494] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7272.134918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7276.522153] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7276.524696] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7276.531258] LustreError: Skipped 2 previous similar messages [ 7276.555201] Lustre: Skipped 4 previous similar messages [ 7276.565173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7276.571610] Lustre: Skipped 5 previous similar messages [ 7288.288129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7288.635713] Lustre: server umount lustre-OST0000 complete [ 7295.972828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7298.497284] Lustre: server umount lustre-MDT0000 complete [ 7302.118610] LustreError: 211734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7302.151082] LustreError: 211734:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 7302.426532] LustreError: 212516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628572 with bad export cookie 2186301178315299112 [ 7302.434148] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7302.439975] LustreError: 212516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7302.795393] Lustre: server umount lustre-MDT0001 complete [ 7306.215662] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 13:16:13 (1788628573) [ 7320.981696] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 7331.289239] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 7363.412508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7363.584235] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7363.639540] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7363.702640] Lustre: lustre-MDT0000: new disk, initializing [ 7363.785467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7363.790831] Lustre: Skipped 2 previous similar messages [ 7363.801599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7368.025699] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7378.205845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7378.326479] Lustre: 218300:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7378.585459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7382.561546] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7390.417136] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7395.551447] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7401.451341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7401.669090] Lustre: 220269:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7403.677362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7406.410886] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7414.039150] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7417.292425] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7417.430935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7420.862513] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7421.030159] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7423.969153] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7423.974650] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7423.985577] Lustre: Skipped 2 previous similar messages [ 7423.992317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7423.999048] Lustre: Skipped 7 previous similar messages [ 7428.843335] Lustre: server umount lustre-OST0000 complete [ 7438.281792] Lustre: server umount lustre-MDT0000 complete [ 7439.841319] LustreError: 218314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7439.857849] LustreError: 218314:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7441.131862] LustreError: 218292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628711 with bad export cookie 2186301178315299994 [ 7441.135405] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7441.147905] LustreError: 218292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7441.363361] Lustre: server umount lustre-MDT0001 complete [ 7447.488475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7451.594613] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7459.594342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7466.082735] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7474.313282] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7478.797455] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7484.845859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7490.355564] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7497.021253] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7507.944177] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7507.956502] LustreError: Skipped 2 previous similar messages [ 7507.968289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7507.970721] Lustre: Skipped 4 previous similar messages [ 7513.033040] Lustre: server umount lustre-OST0000 complete [ 7522.092430] Lustre: server umount lustre-MDT0000 complete [ 7525.152969] LustreError: 222467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628795 with bad export cookie 2186301178315300876 [ 7525.163704] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7525.427567] Lustre: server umount lustre-MDT0001 complete [ 7635.013377] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 7637.556359] Key type lgssc unregistered [ 7637.797877] LNet: 226149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7637.803672] LNetError: 226149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7637.838965] LNet: Removed LNI 192.168.206.108@tcp [ 7638.714166] Key type .llcrypt unregistered [ 7638.716425] Key type ._llcrypt unregistered [ 7660.644126] Key type ._llcrypt registered [ 7660.647240] Key type .llcrypt registered [ 7660.745257] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 7676.067733] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 7676.776056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7676.798200] alg: No test for adler32 (adler32-zlib) [ 7677.876162] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 7678.118520] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 7679.776183] Key type lgssc registered [ 7680.873390] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7712.468336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7712.488107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7713.698495] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7713.725703] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7713.824517] Lustre: lustre-MDT0000: new disk, initializing [ 7713.908133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7713.956390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7718.096844] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7727.805331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7727.914771] Lustre: 229500:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7727.965657] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7727.969199] Lustre: Skipped 1 previous similar message [ 7728.037668] Lustre: lustre-MDT0001: new disk, initializing [ 7728.107945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7728.135533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7728.148770] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7732.164843] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7740.040868] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7744.352111] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7748.880654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7749.082918] Lustre: lustre-OST0000: new disk, initializing [ 7749.085504] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7749.089250] Lustre: 231469:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7749.146905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7753.247406] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7756.319925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7756.339776] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7756.389953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7760.056572] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7763.729652] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7763.915498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7766.967781] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7767.143062] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7771.619277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7771.623143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7771.626074] LustreError: Skipped 1 previous similar message [ 7771.629415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7771.633957] Lustre: Skipped 1 previous similar message [ 7771.646806] Lustre: Skipped 1 previous similar message [ 7774.577490] Lustre: server umount lustre-OST0000 complete [ 7779.299457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7779.307092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7779.318078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7781.866912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7781.868352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7783.996417] Lustre: server umount lustre-MDT0000 complete [ 7787.886982] LustreError: 229492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629057 with bad export cookie 15232507132824621784 [ 7787.899336] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7787.910109] LustreError: 229492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7788.222742] Lustre: server umount lustre-MDT0001 complete [ 7804.214605] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 7812.024339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7812.299244] LustreError: 234401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7812.313769] LustreError: 234401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7812.343691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7815.562591] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7817.703444] LustreError: 234402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7822.818324] LustreError: 234401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7824.105402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7824.483240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7829.846115] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7833.104842] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7839.661154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7839.945879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7845.032496] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7852.184819] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7852.332148] Lustre: lustre-OST0001: new disk, initializing [ 7852.335545] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7852.340603] Lustre: 236709:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7857.177976] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7857.643961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7857.655752] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7857.716067] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7867.268572] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7876.701964] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7886.624980] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 13:25:54 (1788629154) [ 7890.914116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7890.919230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7890.928644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7890.934601] Lustre: Skipped 1 previous similar message [ 7894.861953] Lustre: server umount lustre-MDT0000 complete [ 7897.618049] LustreError: 234384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629167 with bad export cookie 15232507132824622666 [ 7897.625171] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7897.630504] LustreError: 234384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7897.959169] Lustre: server umount lustre-MDT0001 complete [ 7911.739837] Lustre: server umount lustre-OST0000 complete [ 7914.720480] Lustre: 227723:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788629168/real 1788629168] req@ffff9e3e07497100 x1875513387881216/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788629184 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7914.744599] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7914.754455] Lustre: Skipped 3 previous similar messages [ 7914.953624] Lustre: server umount lustre-OST0001 complete [ 7920.368447] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 7928.504872] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 7958.557660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7958.758983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7958.775838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7958.863910] Lustre: lustre-MDT0000: new disk, initializing [ 7958.948105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7958.953151] Lustre: Skipped 1 previous similar message [ 7958.963405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7962.716443] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7972.439657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7972.544782] Lustre: 240653:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7972.610482] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7972.617581] Lustre: Skipped 1 previous similar message [ 7972.699557] Lustre: lustre-MDT0001: new disk, initializing [ 7972.809035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7972.815411] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7977.080520] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7985.766572] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7990.359373] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7996.012758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7996.251557] Lustre: lustre-OST0000: new disk, initializing [ 7996.258666] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7996.266719] Lustre: 242622:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7996.367349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7996.376087] Lustre: Skipped 1 previous similar message [ 7998.286421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7998.293140] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7998.396034] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8002.076905] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8010.228751] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8014.438357] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8014.599164] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8018.164157] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8018.320118] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8024.042790] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8024.047829] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8024.057492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8024.057502] Lustre: Skipped 4 previous similar messages [ 8024.059848] Lustre: Skipped 1 previous similar message [ 8024.094344] LustreError: Skipped 1 previous similar message [ 8025.811141] Lustre: server umount lustre-OST0000 complete [ 8029.155454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8034.272931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8034.279705] Lustre: Skipped 4 previous similar messages [ 8035.107642] Lustre: server umount lustre-MDT0000 complete [ 8037.582485] LustreError: 242621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629307 with bad export cookie 15232507132824624801 [ 8037.588818] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8037.592472] LustreError: 242621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8037.806108] Lustre: server umount lustre-MDT0001 complete [ 8043.719148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8044.060498] LustreError: 244838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8047.281987] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8049.126341] LustreError: 244837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8053.865809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8058.051847] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8064.433897] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8069.039414] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8074.653112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8074.949125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8074.953213] Lustre: Skipped 2 previous similar messages [ 8079.385543] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8084.955853] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8228.836255] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8228.837161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8228.843449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8228.851383] Lustre: Skipped 3 previous similar messages [ 8228.861524] Lustre: Skipped 1 previous similar message [ 8234.361504] Lustre: server umount lustre-OST0000 complete [ 8243.452930] Lustre: server umount lustre-MDT0000 complete [ 8244.200564] LustreError: 244837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8244.218013] LustreError: 244837:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8245.874738] LustreError: 244818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629515 with bad export cookie 15232507132824625683 [ 8245.876914] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8245.880880] LustreError: 244818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8246.098405] Lustre: server umount lustre-MDT0001 complete [ 8352.174056] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 8354.782518] Key type lgssc unregistered [ 8355.119915] LNet: 248443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8355.132772] LNetError: 248443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8355.153841] LNet: Removed LNI 192.168.206.108@tcp [ 8355.996827] Key type .llcrypt unregistered [ 8355.999399] Key type ._llcrypt unregistered [ 8379.372236] Key type ._llcrypt registered [ 8379.374669] Key type .llcrypt registered [ 8379.460396] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 8394.219212] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8395.011152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8395.031826] alg: No test for adler32 (adler32-zlib) [ 8396.197159] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 8396.470188] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 8398.128211] Key type lgssc registered [ 8399.127903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8440.744107] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8449.644993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8449.667801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8450.896649] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8450.928367] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8451.020262] Lustre: lustre-MDT0000: new disk, initializing [ 8451.079369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8451.095596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8454.712899] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8463.564484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8463.661355] Lustre: 252530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8463.701312] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8463.704463] Lustre: Skipped 1 previous similar message [ 8463.759939] Lustre: lustre-MDT0001: new disk, initializing [ 8463.831141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8463.880943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8463.899338] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8468.238083] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8472.505990] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8478.432345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8478.643275] Lustre: lustre-OST0000: new disk, initializing [ 8478.646824] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8478.652934] Lustre: 254162:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8478.757836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8480.362488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8480.369091] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8480.488073] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8484.285882] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8494.249958] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8494.429955] Lustre: lustre-OST0001: new disk, initializing [ 8494.436535] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8494.446622] Lustre: 255033:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8494.558627] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8499.903240] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8503.871766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8503.880558] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8503.950929] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8510.008259] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8516.717906] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8529.895867] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8529.901358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8529.908139] Lustre: Skipped 1 previous similar message [ 8529.923839] Lustre: Skipped 3 previous similar messages [ 8532.755258] Lustre: server umount lustre-MDT0000 complete [ 8536.056971] LustreError: 255799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629806 with bad export cookie 1678664967623373673 [ 8536.063307] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8536.074605] LustreError: 255799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8536.421978] Lustre: server umount lustre-MDT0001 complete [ 8549.469461] Lustre: server umount lustre-OST0000 complete [ 8562.231619] Lustre: server umount lustre-OST0001 complete [ 8573.952700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8582.334978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8590.260134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8598.407791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8614.994612] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8623.587966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8623.606353] Lustre: lustre-MDT0000: reset Object Index mappings [ 8624.019088] LustreError: 260041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8624.039089] LustreError: 260041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 8624.084038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8627.996402] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8629.225029] LustreError: 260042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8634.348943] LustreError: 260041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8634.739316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8634.765822] Lustre: lustre-MDT0001: reset Object Index mappings [ 8635.146978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8638.814652] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8641.253919] Lustre: 261182:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8646.883302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8649.210431] LustreError: 261536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8649.243660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8652.049959] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8654.311925] LustreError: 261536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8659.197345] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8659.434944] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8659.439915] Lustre: Skipped 1 previous similar message [ 8660.485113] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 8664.641991] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8671.920706] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8675.390885] Lustre: 263055:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8680.940059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8680.942447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8680.961097] Lustre: Skipped 4 previous similar messages [ 8680.966168] Lustre: Skipped 3 previous similar messages [ 8682.970102] Lustre: server umount lustre-MDT0000 complete [ 8685.772126] LustreError: 263159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629955 with bad export cookie 1678664967623380463 [ 8685.780482] LustreError: 263159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8685.780633] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8686.091982] Lustre: server umount lustre-MDT0001 complete [ 8699.714930] Lustre: server umount lustre-OST0000 complete [ 8702.625102] Lustre: 250037:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788629956/real 1788629956] req@ffff9e3e0f75ce00 x1875514140774912/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788629972 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8702.646457] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8702.656716] Lustre: Skipped 1 previous similar message [ 8702.802178] Lustre: server umount lustre-OST0001 complete [ 8717.130393] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 13:39:44 (1788629984) [ 8728.731582] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 8737.164500] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8767.136771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8767.338263] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8767.374808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8767.465437] Lustre: lustre-MDT0000: new disk, initializing [ 8767.532051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8767.543412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8771.269363] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8779.957200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8780.075906] Lustre: 267309:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8780.126764] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8780.130364] Lustre: Skipped 1 previous similar message [ 8780.208496] Lustre: lustre-MDT0001: new disk, initializing [ 8780.327834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8780.342380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8783.984315] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8791.995861] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8797.270716] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8803.619369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8803.898685] Lustre: lustre-OST0000: new disk, initializing [ 8803.901985] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8803.909168] Lustre: 269276:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8805.307338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8805.317423] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8805.402740] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8809.671812] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8817.680326] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8821.800147] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8821.966512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8826.278781] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8826.528259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8830.947662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8830.949886] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8830.950209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8830.955662] Lustre: Skipped 1 previous similar message [ 8830.971746] LustreError: Skipped 1 previous similar message [ 8834.516451] Lustre: server umount lustre-OST0000 complete [ 8841.188644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8841.189572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8841.203272] Lustre: Skipped 1 previous similar message [ 8841.216817] Lustre: Skipped 1 previous similar message [ 8844.175538] Lustre: server umount lustre-MDT0000 complete [ 8846.306520] LustreError: 268194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8846.318373] LustreError: 268194:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8847.400359] LustreError: 267300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630117 with bad export cookie 1678664967623382556 [ 8847.404451] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8847.410548] LustreError: 267300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8847.816154] Lustre: server umount lustre-MDT0001 complete [ 8861.604821] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8869.037972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8869.334510] LustreError: 272207:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8869.378695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8869.386152] Lustre: Skipped 2 previous similar messages [ 8872.556814] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8879.944165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8884.316472] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8886.951132] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8893.546986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8900.239677] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8907.916962] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8908.200067] Lustre: lustre-OST0001: new disk, initializing [ 8908.207619] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8908.212640] Lustre: 274514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8909.844211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8909.856529] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8909.974929] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8913.918898] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8922.348095] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8925.676054] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8930.512852] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 8930.612921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 8940.523172] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8940.527857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8940.537617] Lustre: Skipped 1 previous similar message [ 8940.550891] Lustre: Skipped 3 previous similar messages [ 8944.260967] Lustre: server umount lustre-MDT0000 complete [ 8945.641433] LustreError: 272208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8945.693071] LustreError: 272208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 8947.455830] LustreError: 272187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630217 with bad export cookie 1678664967623383438 [ 8947.462077] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8947.467319] LustreError: 272187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8947.753924] Lustre: server umount lustre-MDT0001 complete [ 8961.088528] Lustre: server umount lustre-OST0000 complete [ 8973.373892] Lustre: server umount lustre-OST0001 complete [ 8987.508341] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8995.422621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9000.006701] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9008.225279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9008.521694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9008.524445] Lustre: Skipped 4 previous similar messages [ 9012.984937] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9015.507413] Lustre: 279002:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9021.517768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9028.076867] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9033.129585] LustreError: 279356:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9033.142824] LustreError: 279356:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9035.284678] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9040.643501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9040.659413] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9041.152085] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9048.413894] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9051.731447] Lustre: 280878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9056.340142] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9056.368655] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9067.365316] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 13:45:34 (1788630334) [ 9084.901060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9084.911195] Lustre: Skipped 2 previous similar messages [ 9084.916280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9084.931841] Lustre: Skipped 3 previous similar messages [ 9087.462962] Lustre: server umount lustre-MDT0000 complete [ 9090.650948] LustreError: 278602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630360 with bad export cookie 1678664967623390039 [ 9090.652758] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9090.659946] LustreError: 278602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9096.981411] Lustre: server umount lustre-MDT0001 complete [ 9105.866565] Lustre: server umount lustre-OST0000 complete [ 9123.296285] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9123.462820] Lustre: server umount lustre-OST0001 complete [ 9128.624032] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 9138.363470] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 9179.239894] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 9188.839916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9189.173307] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9189.232912] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9189.322122] Lustre: lustre-MDT0000: new disk, initializing [ 9189.451479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9193.769966] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9202.580327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9202.721420] Lustre: 285406:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9202.998541] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9206.810432] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9217.072469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9217.314936] Lustre: 287038:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9218.639792] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9222.819532] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9231.696558] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9231.888133] Lustre: 287909:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9233.785794] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9237.353661] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9245.539605] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9248.303620] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9248.307574] Lustre: Skipped 1 previous similar message [ 9259.945947] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 13:48:47 (1788630527) [ 9264.617744] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9264.629837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9264.630819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9264.642896] Lustre: Skipped 4 previous similar messages [ 9264.653815] Lustre: Skipped 5 previous similar messages [ 9268.104128] Lustre: server umount lustre-MDT0000 complete [ 9269.741884] LustreError: 286310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9269.775156] LustreError: 286310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 9271.793045] LustreError: 285399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630541 with bad export cookie 1678664967623397774 [ 9271.796096] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9271.801892] LustreError: 285399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9272.057768] Lustre: server umount lustre-MDT0001 complete [ 9285.839289] Lustre: server umount lustre-OST0000 complete [ 9298.689039] Lustre: server umount lustre-OST0001 complete [ 9304.937118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9305.338805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9305.349197] Lustre: Skipped 6 previous similar messages [ 9308.678412] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9314.695142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9318.128748] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9323.764861] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9327.034785] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9331.537343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9336.728878] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9343.696228] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9366.421603] loop1: detected capacity change from 0 to 204800000 [ 9366.506134] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9366.511046] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9366.516312] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9371.043029] loop1: detected capacity change from 0 to 204800000 [ 9375.584466] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9375.673492] LustreError: 290144:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs16155: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 9375.685214] LustreError: 290144:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs16155 [ 9375.690866] LustreError: 290144:0:(mgs_handler.c:517:mgs_target_reg()) fs16155-MDT0001: Error checking target index: rc = -13 [ 9375.700768] LustreError: fs16155-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9375.708419] LustreError: 293892:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [ 9375.715746] LustreError: 293892:0:(tgt_mount.c:2132:server_put_super()) no obd fs16155-MDT0001 [ 9375.719945] LustreError: 293892:0:(tgt_mount.c:116:server_deregister_mount()) fs16155-MDT0001 not registered [ 9375.763762] Lustre: server umount fs16155-MDT0001 complete [ 9375.767140] LustreError: 293892:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9382.551938] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9382.620435] Lustre: Setting parameter fs16155-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs16155-MDT0001 [ 9382.679567] Lustre: srv-fs16155-MDT0001: No data found on store. Initialize space. [ 9382.684454] Lustre: Skipped 4 previous similar messages [ 9382.754770] Lustre: fs16155-MDT0001: new disk, initializing [ 9382.758494] Lustre: Skipped 3 previous similar messages [ 9382.856600] Lustre: srv-fs16155-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9382.862973] Lustre: 294360:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16155-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9382.871511] LustreError: 294360:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16155-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9386.083250] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9391.847891] Lustre: server umount fs16155-MDT0001 complete [ 9402.134643] loop1: detected capacity change from 0 to 204800000 [ 9405.994456] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9406.321083] Lustre: 295928:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16155-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9406.329049] LustreError: 295928:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16155-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9409.813230] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9413.567128] Lustre: server umount fs16155-MDT0001 complete [ 9421.223198] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.8@tcp (stopping) [ 9422.305370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9422.316516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9422.331164] Lustre: Skipped 1 previous similar message [ 9426.949794] Lustre: server umount lustre-MDT0000 complete [ 9435.180771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9435.424907] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9440.403802] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9441.277820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9447.614624] loop1: detected capacity change from 0 to 204800000 [ 9452.924530] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9453.167654] Lustre: 298372:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16155-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9453.175865] LustreError: 298372:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16155-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9457.595378] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9461.769360] Lustre: server umount fs16155-MDT0001 complete [ 9486.819629] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9486.826777] LustreError: Skipped 1 previous similar message [ 9486.831560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9486.837812] Lustre: Skipped 4 previous similar messages [ 9490.318184] Lustre: server umount lustre-OST0000 complete [ 9497.586779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9499.707761] Lustre: server umount lustre-MDT0000 complete [ 9502.992927] LustreError: 290137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630773 with bad export cookie 1678664967623400455 [ 9502.996145] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9503.011541] LustreError: 290137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9503.458585] Lustre: server umount lustre-MDT0001 complete [ 9513.490869] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 9516.306049] Key type lgssc unregistered [ 9516.612879] LNet: 300714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9516.630606] LNetError: 300714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9516.675742] LNet: Removed LNI 192.168.206.108@tcp [ 9517.500233] Key type .llcrypt unregistered [ 9517.503318] Key type ._llcrypt unregistered [ 9536.662206] Key type ._llcrypt registered [ 9536.664393] Key type .llcrypt registered [ 9536.788790] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 9538.158518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9538.192323] alg: No test for adler32 (adler32-zlib) [ 9539.206805] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 9539.501363] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 9541.144176] Key type lgssc registered [ 9542.379442] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9550.111157] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9550.131991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9551.723128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9556.294430] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9561.069249] LustreError: 301986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9561.105697] LustreError: 301986:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9564.724096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9565.184765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9568.880618] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9576.055643] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9582.631633] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9590.986388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9591.653592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9596.804326] LustreError: 303814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9599.942565] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9602.032123] LustreError: 303817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9607.143032] LustreError: 304005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9608.558374] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9611.280366] LustreError: 303814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9611.317668] LustreError: 303814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9620.457744] LustreError: 304005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9620.480835] LustreError: 304005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9626.554624] loop1: detected capacity change from 0 to 204800000 [ 9626.721172] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9626.735222] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9626.747087] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9632.078789] loop1: detected capacity change from 0 to 204800000 [ 9637.258527] LustreError: 303816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9637.274717] LustreError: 303816:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 9637.976706] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9638.117703] Lustre: fs18819-OST0000: new disk, initializing [ 9638.120960] Lustre: srv-fs18819-OST0000: No data found on store. Initialize space. [ 9638.126908] Lustre: 305722:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9638.199644] Lustre: fs18819-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9643.775894] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9653.118455] Lustre: server umount fs18819-OST0000 complete [ 9666.568399] loop1: detected capacity change from 0 to 204800000 [ 9671.668227] LustreError: 304005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9671.692147] LustreError: 304005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 9672.512557] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9672.789599] Lustre: fs18819-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9676.776901] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 9678.874399] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9687.274128] loop2: detected capacity change from 0 to 204800000 [ 9687.532354] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9687.553305] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9687.561383] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9692.138045] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 9693.083679] loop2: detected capacity change from 0 to 204800000 [ 9698.532262] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9698.648622] LustreError: fs18819-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 9698.660371] LustreError: fs18819-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9698.671268] LustreError: 308762:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [ 9698.681521] LustreError: 308762:0:(tgt_mount.c:2132:server_put_super()) no obd fs18819-OST0001 [ 9698.692320] LustreError: 308762:0:(tgt_mount.c:116:server_deregister_mount()) fs18819-OST0001 not registered [ 9698.749956] Lustre: server umount fs18819-OST0001 complete [ 9698.753960] LustreError: 308762:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9708.928176] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9709.134248] Lustre: fs18819-OST0001: new disk, initializing [ 9709.138280] Lustre: srv-fs18819-OST0001: No data found on store. Initialize space. [ 9709.144817] Lustre: 309314:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9709.245995] Lustre: fs18819-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9715.167744] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9723.417108] Lustre: server umount fs18819-OST0000 complete [ 9731.633325] Lustre: server umount fs18819-OST0001 complete [ 9738.218682] LustreError: 303814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9738.240630] LustreError: 303814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [ 9743.665301] loop1: detected capacity change from 0 to 204800000 [ 9749.044405] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9749.483698] Lustre: fs18819-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9756.404776] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9762.984513] loop2: detected capacity change from 0 to 204800000 [ 9768.602219] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9774.776035] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9785.090154] Lustre: server umount fs18819-OST0000 complete [ 9793.331705] Lustre: server umount fs18819-OST0001 complete [ 9806.817309] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9806.829254] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9806.845463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9809.898915] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9809.900563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9809.922869] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9809.942272] Lustre: Skipped 1 previous similar message [ 9811.872928] Lustre: server umount lustre-OST0000 complete [ 9816.038235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9816.053498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9816.066495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9820.131749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9820.133492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9820.162398] Lustre: Skipped 1 previous similar message [ 9822.125734] Lustre: server umount lustre-MDT0000 complete [ 9826.085859] LustreError: 301966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631096 with bad export cookie 18020517038895269740 [ 9826.096606] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9826.106867] LustreError: 301966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9826.528811] Lustre: server umount lustre-MDT0001 complete [ 9836.997714] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 9839.808261] Key type lgssc unregistered [ 9840.111123] LNet: 315236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9840.129616] LNetError: 315236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9841.196399] LNet: Removed LNI 192.168.206.108@tcp [ 9842.242048] Key type .llcrypt unregistered [ 9842.243784] Key type ._llcrypt unregistered [ 9872.443461] Key type ._llcrypt registered [ 9872.444684] Key type .llcrypt registered [ 9872.535520] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 9887.697348] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 9888.853992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9888.868776] alg: No test for adler32 (adler32-zlib) [ 9890.025816] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 9890.334167] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 9892.096149] Key type lgssc registered [ 9893.347450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9930.223720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9930.284661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9931.458203] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9931.484628] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9931.544870] Lustre: lustre-MDT0000: new disk, initializing [ 9931.608774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9931.642458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9936.880895] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9946.695521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9946.906766] Lustre: 319091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9947.024949] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9947.029767] Lustre: Skipped 1 previous similar message [ 9947.196509] Lustre: lustre-MDT0001: new disk, initializing [ 9947.352331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9947.413984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9947.434149] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9952.524432] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9963.212805] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9969.133976] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9974.906655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9975.158762] Lustre: lustre-OST0000: new disk, initializing [ 9975.164198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9975.170176] Lustre: 321064:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9975.253412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9981.275768] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9981.535137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9981.550587] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9981.665138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9990.492075] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9992.804197] Lustre: Mounted lustre-client - version 2.17.58_2_gf69a9da [ 9993.807176] Lustre: Unmounted lustre-client [10007.521317] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10007.522984] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10007.532986] LustreError: Skipped 1 previous similar message [10007.544970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10007.567372] Lustre: Skipped 1 previous similar message [10009.498167] Lustre: server umount lustre-OST0000 complete [10014.177545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10014.191056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10014.210198] Lustre: Skipped 1 previous similar message [10014.216197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10014.218946] Lustre: Skipped 2 previous similar messages [10018.273972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10018.277159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10019.482571] Lustre: server umount lustre-MDT0000 complete [10023.322616] LustreError: 319897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631293 with bad export cookie 11941842816878112992 [10023.340790] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10023.660806] Lustre: server umount lustre-MDT0001 complete [10034.702135] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [10037.521669] Key type lgssc unregistered [10047.074664] LNet: 323234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10047.092486] LNetError: 323234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10122.919207] LNet: Removed LNI 192.168.206.108@tcp [10123.633087] Key type .llcrypt unregistered [10123.638066] Key type ._llcrypt unregistered [10141.276823] Key type ._llcrypt registered [10141.278903] Key type .llcrypt registered [10141.399668] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [10155.757480] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [10157.000635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10157.046831] alg: No test for adler32 (adler32-zlib) [10158.120345] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [10158.389317] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [10160.056483] Key type lgssc registered [10161.137422] Lustre: Echo OBD driver; http://www.lustre.org/ [10188.708760] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 14:04:16 (1788631456) [10194.917706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10194.947410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10196.143713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10196.184516] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10196.254362] Lustre: lustre-MDT0000: new disk, initializing [10196.347277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10196.367427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10200.136705] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10208.877180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10208.964063] Lustre: 326455:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10209.009483] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10209.013639] Lustre: Skipped 1 previous similar message [10209.103337] Lustre: lustre-MDT0001: new disk, initializing [10209.205705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10209.241603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10209.255637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10214.219315] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10222.824266] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10228.008852] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10234.721630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10234.938932] Lustre: lustre-OST0000: new disk, initializing [10234.942161] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10234.947875] Lustre: 328423:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10235.024951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10241.160731] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10241.643618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10241.657975] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10241.836996] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10252.680375] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10267.619407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10267.622947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10267.638415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10267.653691] Lustre: Skipped 1 previous similar message [10267.669147] Lustre: Skipped 1 previous similar message [10272.742714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10272.746928] Lustre: Skipped 1 previous similar message [10273.235511] Lustre: server umount lustre-OST0000 complete [10277.865706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10277.867178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10277.869579] Lustre: Skipped 1 previous similar message [10282.832694] Lustre: server umount lustre-MDT0000 complete [10286.273054] LustreError: 326446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631556 with bad export cookie 13193665509686760894 [10286.277034] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10286.290509] LustreError: 326446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10286.572580] Lustre: server umount lustre-MDT0001 complete [10297.124370] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [10300.059052] Key type lgssc unregistered [10300.325553] LNet: 330329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10300.342673] LNetError: 330329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10300.368636] LNet: Removed LNI 192.168.206.108@tcp [10301.333506] Key type .llcrypt unregistered [10301.335256] Key type ._llcrypt unregistered [10321.896652] Key type ._llcrypt registered [10321.902484] Key type .llcrypt registered [10322.029710] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [10322.819788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10322.903417] alg: No test for adler32 (adler32-zlib) [10324.000309] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [10324.248253] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [10326.008604] Key type lgssc registered [10327.244997] Lustre: Echo OBD driver; http://www.lustre.org/ [10334.133619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10334.168943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10335.705521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10340.049671] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10345.955220] LustreError: 331600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10345.983361] LustreError: 331600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10348.019589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10348.378697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10352.853317] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10359.660403] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10364.921842] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10370.331818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10370.582940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10375.418687] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10382.053317] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10390.608506] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10391.888966] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10396.134215] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10396.139865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10396.145031] Lustre: Skipped 2 previous similar messages [10396.156593] Lustre: Skipped 2 previous similar messages [10399.150838] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.8@tcp (stopping) [10401.252179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10401.266383] Lustre: Skipped 2 previous similar messages [10401.680869] Lustre: server umount lustre-MDT0000 complete [10404.243140] LustreError: 331595:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10405.039887] LustreError: 331581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631675 with bad export cookie 10166676605871100065 [10405.046535] LustreError: 331581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10405.048873] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10406.369758] LustreError: 331597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10406.371093] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10406.409833] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10409.360799] LustreError: 331595:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10411.420958] Lustre: server umount lustre-MDT0001 complete [10414.777654] Lustre: server umount lustre-OST0000 complete [10422.533681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10427.041411] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10429.617643] Lustre: 335779:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.108@tcp [10431.464947] Lustre: server umount MGS complete [10439.477453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10439.918975] LustreError: 336354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10439.947229] LustreError: 336354:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10439.999959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10444.316290] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10449.288381] LustreError: 336349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10449.304529] LustreError: 336349:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10450.887261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10451.329282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10455.362241] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10464.442196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10464.818460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10470.608802] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10477.431171] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10478.822391] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [10484.092866] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10485.370641] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10490.345117] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10490.354158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10490.369430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10490.374366] Lustre: Skipped 2 previous similar messages [10491.375508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10491.378675] LustreError: 337790:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10491.410376] LustreError: 337790:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10491.504423] Lustre: server umount lustre-OST0000 complete [10496.484136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10496.492765] Lustre: Skipped 2 previous similar messages [10500.804184] Lustre: server umount lustre-MDT0000 complete [10504.118305] LustreError: 336334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631774 with bad export cookie 10166676605871100905 [10504.129086] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10504.130690] LustreError: 336334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10504.425856] Lustre: server umount lustre-MDT0001 complete [10513.449268] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [10516.008947] Key type lgssc unregistered [10516.322807] LNet: 339787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10516.334315] LNetError: 339787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10516.368763] LNet: Removed LNI 192.168.206.108@tcp [10517.226125] Key type .llcrypt unregistered [10517.227671] Key type ._llcrypt unregistered [10533.440156] Key type ._llcrypt registered [10533.442547] Key type .llcrypt registered [10533.524248] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [10548.183879] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [10549.090661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10549.257761] alg: No test for adler32 (adler32-zlib) [10550.321393] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [10550.568167] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [10552.216568] Key type lgssc registered [10553.329764] Lustre: Echo OBD driver; http://www.lustre.org/ [10583.971756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10584.012570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10585.276629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10585.299253] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10585.414365] Lustre: lustre-MDT0000: new disk, initializing [10585.587546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10585.614863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10590.005208] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10598.931222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10599.059933] Lustre: 342943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10599.129717] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10599.133831] Lustre: Skipped 1 previous similar message [10599.208945] Lustre: lustre-MDT0001: new disk, initializing [10599.284465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10599.318896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10599.327496] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10602.439793] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10609.558438] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10614.486489] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10621.063730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10621.327689] Lustre: lustre-OST0000: new disk, initializing [10621.335847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10621.348450] Lustre: 344911:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10621.501756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10627.222933] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10628.196859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10628.208831] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10628.334811] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10635.887554] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10641.038367] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10641.318374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10646.304901] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10646.614176] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10649.060659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10649.070116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10649.081609] Lustre: Skipped 1 previous similar message [10649.083732] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10654.184251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10654.188725] Lustre: Skipped 1 previous similar message [10654.741603] Lustre: server umount lustre-OST0000 complete [10659.303888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10659.313794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10659.330059] Lustre: Skipped 1 previous similar message [10659.333083] Lustre: Skipped 1 previous similar message [10664.423173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10664.429145] Lustre: Skipped 2 previous similar messages [10665.071676] Lustre: server umount lustre-MDT0000 complete [10668.951038] LustreError: 344910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631938 with bad export cookie 14565901913262098675 [10668.954876] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10668.958401] LustreError: 344910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10669.562991] Lustre: server umount lustre-MDT0001 complete [10678.549812] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 14:12:25 (1788631945) [10693.892607] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [10703.467251] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [10735.024695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10735.203568] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10735.318044] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10735.441913] Lustre: lustre-MDT0000: new disk, initializing [10735.535774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10735.563287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10739.619152] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10750.028363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10750.107760] Lustre: 349669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10750.151845] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10750.154972] Lustre: Skipped 1 previous similar message [10750.234848] Lustre: lustre-MDT0001: new disk, initializing [10750.309781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10750.351090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10750.367344] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10754.411897] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10763.268443] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10769.346692] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10775.320630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10775.547671] Lustre: lustre-OST0000: new disk, initializing [10775.550367] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10775.556933] Lustre: 351636:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10775.623978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10781.087296] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10783.843493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10783.855758] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10783.989338] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10789.661832] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10820.072424] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10820.077562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10820.078384] LustreError: Skipped 1 previous similar message [10820.083592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10820.099312] Lustre: Skipped 1 previous similar message [10823.285836] Lustre: server umount lustre-OST0000 complete [10827.233967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10827.242551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10830.305110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10830.312167] Lustre: Skipped 3 previous similar messages [10832.778099] Lustre: server umount lustre-MDT0000 complete [10835.606719] LustreError: 349661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788632105 with bad export cookie 14565901913262099557 [10835.612762] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10835.631784] LustreError: 349661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10835.944987] Lustre: server umount lustre-MDT0001 complete [10843.964377] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [10846.349829] Key type lgssc unregistered [10846.607627] LNet: 353788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10846.615225] LNetError: 353788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10846.631167] LNet: Removed LNI 192.168.206.108@tcp [10847.588885] Key type .llcrypt unregistered [10847.593676] Key type ._llcrypt unregistered [10918.800274] Key type ._llcrypt registered [10918.802415] Key type .llcrypt registered [10918.889423] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [10919.179072] Key type .llcrypt unregistered [10919.180581] Key type ._llcrypt unregistered [10984.218515] Key type ._llcrypt registered [10984.221628] Key type .llcrypt registered [10984.315954] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [10984.567025] Key type .llcrypt unregistered [10984.570127] Key type ._llcrypt unregistered [11007.452093] Key type ._llcrypt registered [11007.455027] Key type .llcrypt registered [11007.535121] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11007.781520] Key type .llcrypt unregistered [11007.784210] Key type ._llcrypt unregistered [11029.774161] Key type ._llcrypt registered [11029.775784] Key type .llcrypt registered [11029.914239] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11030.152677] Key type .llcrypt unregistered [11030.156947] Key type ._llcrypt unregistered [11094.483453] Key type ._llcrypt registered [11094.488341] Key type .llcrypt registered [11094.620805] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11095.021414] Key type .llcrypt unregistered [11095.026792] Key type ._llcrypt unregistered [11120.698834] Key type ._llcrypt registered [11120.703750] Key type .llcrypt registered [11120.784971] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11121.003556] Key type .llcrypt unregistered [11121.007585] Key type ._llcrypt unregistered [11144.216189] Key type ._llcrypt registered [11144.227591] Key type .llcrypt registered [11144.389303] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11144.659431] Key type .llcrypt unregistered [11144.662668] Key type ._llcrypt unregistered [11170.386168] Key type ._llcrypt registered [11170.387764] Key type .llcrypt registered [11170.483200] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11170.701134] Key type .llcrypt unregistered [11170.703765] Key type ._llcrypt unregistered [11197.258505] Key type ._llcrypt registered [11197.263519] Key type .llcrypt registered [11197.389289] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11197.649563] Key type .llcrypt unregistered [11197.654317] Key type ._llcrypt unregistered [11228.353325] Key type ._llcrypt registered [11228.355704] Key type .llcrypt registered [11228.434969] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11228.621369] Key type .llcrypt unregistered [11228.626239] Key type ._llcrypt unregistered [11279.899634] Key type ._llcrypt registered [11279.905380] Key type .llcrypt registered [11280.004680] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11280.223309] Key type .llcrypt unregistered [11280.225322] Key type ._llcrypt unregistered [11306.057079] Key type ._llcrypt registered [11306.058373] Key type .llcrypt registered [11306.154527] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11306.385745] Key type .llcrypt unregistered [11306.387360] Key type ._llcrypt unregistered [11332.577893] Key type ._llcrypt registered [11332.580215] Key type .llcrypt registered [11332.679578] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11333.061364] Key type .llcrypt unregistered [11333.066108] Key type ._llcrypt unregistered [11375.699850] Key type ._llcrypt registered [11375.705730] Key type .llcrypt registered [11375.843613] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [11390.006952] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [11391.506398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11391.553769] alg: No test for adler32 (adler32-zlib) [11392.745981] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [11393.065271] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [11394.808190] Key type lgssc registered [11396.036524] Lustre: Echo OBD driver; http://www.lustre.org/ [11425.759673] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11425.786653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11427.074114] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [11427.094154] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [11427.158601] Lustre: tst250fs-MDT0000: new disk, initializing [11427.223360] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11427.244244] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11430.840203] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [11439.675228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11439.764430] Lustre: 373746:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11439.821340] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [11439.824728] Lustre: Skipped 1 previous similar message [11439.930388] Lustre: tst250fs-MDT0001: new disk, initializing [11440.071227] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11440.109482] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11440.126925] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11444.363259] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [11448.275411] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11453.223561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11453.449811] Lustre: tst250fs-OST0000: new disk, initializing [11453.454774] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [11453.468397] Lustre: 375378:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11453.568587] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11457.472743] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11457.484894] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11457.573550] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11458.734029] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [11468.000517] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11468.119117] Lustre: tst250fs-OST0001: new disk, initializing [11468.123977] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [11468.133152] Lustre: 376247:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11468.228427] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11473.963578] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [11476.531991] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11476.540187] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11476.639161] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11478.908090] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [11481.262634] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [11481.265967] Lustre: Skipped 2 previous similar messages [11501.544621] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11501.550301] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11501.574691] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11502.064121] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11502.081254] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11502.089336] Lustre: Skipped 2 previous similar messages [11502.101953] Lustre: Skipped 3 previous similar messages [11507.175590] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11507.190220] Lustre: Skipped 3 previous similar messages [11512.288409] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11512.303933] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11512.321349] Lustre: Skipped 3 previous similar messages [11512.595450] Lustre: server umount tst250fs-MDT0000 complete [11516.676686] LustreError: 375009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788632786 with bad export cookie 6954706870293026073 [11516.688917] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11516.711973] LustreError: 375009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11517.407227] Lustre: server umount tst250fs-MDT0001 complete [11531.275710] Lustre: server umount tst250fs-OST0000 complete [11533.280118] Lustre: 371972:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788632787/real 1788632787] req@ffff9e3e0e4e1500 x1875517283304832/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788632803 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11533.314947] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11534.925491] Lustre: server umount tst250fs-OST0001 complete [11554.375828] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [11563.941898] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [11592.971476] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 14:27:40 (1788632860) [11594.340152] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [11611.476363] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [11621.254535] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [11652.881815] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11395 sec ========= 14:28:39 (1788632919) [11654.901267] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:28:42 (1788632922) === [11658.185288] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:28:45 (1788632925) === [11690.195306] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11692.565871] Key type lgssc unregistered [11692.873619] LNet: 385390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11692.879977] LNetError: 385390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11692.897935] LNet: Removed LNI 192.168.206.108@tcp [11693.643129] Key type .llcrypt unregistered [11693.646731] Key type ._llcrypt unregistered