[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 470427971 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002301] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007648] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.011053] LSM: Security Framework initializing [ 0.012053] Yama: becoming mindful. [ 0.013033] SELinux: Initializing. [ 0.014059] *** VALIDATE selinux *** [ 0.022172] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027259] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028140] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030043] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031110] *** VALIDATE tmpfs *** [ 0.033197] *** VALIDATE proc *** [ 0.034242] *** VALIDATE cgroup *** [ 0.035008] *** VALIDATE cgroup2 *** [ 0.036247] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037141] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039025] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.042728] debug: unmapping init [mem 0xffffffff89859000-0xffffffff89860fff] [ 0.045136] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046729] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047023] ... version: 2 [ 0.048011] ... bit width: 48 [ 0.049010] ... generic registers: 4 [ 0.050011] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052013] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054314] rcu: Hierarchical SRCU implementation. [ 0.056393] smp: Bringing up secondary CPUs ... [ 0.057589] x86: Booting SMP configuration: [ 0.058024] .... node #0, CPUs: #1 #2 #3 [ 0.067081] smp: Brought up 1 node, 4 CPUs [ 0.069017] smpboot: Max logical packages: 1 [ 0.070012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.161842] node 0 deferred pages initialised in 90ms [ 0.166249] devtmpfs: initialized [ 0.168227] x86/mm: Memory block size: 128MB [ 0.170729] gcov: version magic: 0x41383552 [ 0.173184] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.177073] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.179252] pinctrl core: initialized pinctrl subsystem [ 0.181120] [ 0.181423] ************************************************************* [ 0.183011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.185012] ** ** [ 0.188010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.190011] ** ** [ 0.193012] ** This means that this kernel is built to expose internal ** [ 0.195011] ** IOMMU data structures, which may compromise security on ** [ 0.197012] ** your system. ** [ 0.200012] ** ** [ 0.202012] ** If you see this message and you are not debugging the ** [ 0.204043] ** kernel, report this immediately to your vendor! ** [ 0.205008] ** ** [ 0.207018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.208035] ************************************************************* [ 0.211093] NET: Registered protocol family 16 [ 0.212369] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.215049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.218067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.223010] cpuidle: using governor menu [ 0.224507] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.226473] PCI: Using configuration type 1 for base access [ 0.228119] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.238123] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.240021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.243030] cryptd: max_cpu_qlen set to 1000 [ 0.245565] ACPI: Added _OSI(Module Device) [ 0.248014] ACPI: Added _OSI(Processor Device) [ 0.250017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.251010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.255561] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.260333] ACPI: Interpreter enabled [ 0.261040] ACPI: PM: (supports S0 S3 S4 S5) [ 0.261911] ACPI: Using IOAPIC for interrupt routing [ 0.263103] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.265264] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.274175] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.276036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.279018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.282097] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.287273] acpiphp: Slot [2] registered [ 0.289144] acpiphp: Slot [5] registered [ 0.291127] acpiphp: Slot [6] registered [ 0.292103] acpiphp: Slot [7] registered [ 0.294120] acpiphp: Slot [8] registered [ 0.295070] acpiphp: Slot [9] registered [ 0.296012] acpiphp: Slot [10] registered [ 0.297063] acpiphp: Slot [3] registered [ 0.298077] acpiphp: Slot [4] registered [ 0.299011] acpiphp: Slot [11] registered [ 0.300032] acpiphp: Slot [12] registered [ 0.301067] acpiphp: Slot [13] registered [ 0.303108] acpiphp: Slot [14] registered [ 0.304065] acpiphp: Slot [15] registered [ 0.305065] acpiphp: Slot [16] registered [ 0.307121] acpiphp: Slot [17] registered [ 0.308086] acpiphp: Slot [18] registered [ 0.309149] acpiphp: Slot [19] registered [ 0.311104] acpiphp: Slot [20] registered [ 0.313100] acpiphp: Slot [21] registered [ 0.314091] acpiphp: Slot [22] registered [ 0.316107] acpiphp: Slot [23] registered [ 0.318131] acpiphp: Slot [24] registered [ 0.319096] acpiphp: Slot [25] registered [ 0.320093] acpiphp: Slot [26] registered [ 0.322101] acpiphp: Slot [27] registered [ 0.323095] acpiphp: Slot [28] registered [ 0.325103] acpiphp: Slot [29] registered [ 0.326180] acpiphp: Slot [30] registered [ 0.328100] acpiphp: Slot [31] registered [ 0.329071] PCI host bridge to bus 0000:00 [ 0.330013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.332022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.334020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.336014] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.338018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.339016] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.341152] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.342680] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.343851] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.352012] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.356817] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.359019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.361016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.363016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.365428] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.370964] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.374047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.377809] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.382016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.398017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.404019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.410455] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.423017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.437026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.473017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.484436] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.494014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.503021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.524019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.535760] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.549019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.566018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.598026] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.634014] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.643016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.652020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.671018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.683881] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.690020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.698019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.715020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.726955] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.753024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.760019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.779015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.792489] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.795425] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.798382] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.801402] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.803299] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.808061] iommu: Default domain type: Passthrough [ 0.809344] SCSI subsystem initialized [ 0.810140] ACPI: bus type USB registered [ 0.812095] usbcore: registered new interface driver usbfs [ 0.814095] usbcore: registered new interface driver hub [ 0.816077] usbcore: registered new device driver usb [ 0.818172] pps_core: LinuxPPS API ver. 1 registered [ 0.820011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.822061] PTP clock support registered [ 0.825059] EDAC MC: Ver: 3.0.0 [ 0.827134] PCI: Using ACPI for IRQ routing [ 0.828834] NetLabel: Initializing [ 0.829008] NetLabel: domain hash size = 128 [ 0.830007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.831135] NetLabel: unlabeled traffic allowed by default [ 0.834103] vgaarb: loaded [ 0.835277] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.837014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.843343] clocksource: Switched to clocksource kvm-clock [ 0.968904] VFS: Disk quotas dquot_6.6.0 [ 0.970215] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.972387] *** VALIDATE ramfs *** [ 0.973449] *** VALIDATE hugetlbfs *** [ 0.974762] pnp: PnP ACPI init [ 0.977076] pnp: PnP ACPI: found 6 devices [ 0.993493] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.997155] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.999629] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.002038] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.004556] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.006797] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.009625] NET: Registered protocol family 2 [ 1.011968] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.016881] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.020824] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.026600] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.030507] TCP: Hash tables configured (established 65536 bind 65536) [ 1.033689] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.037215] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.040371] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.043711] NET: Registered protocol family 1 [ 1.046768] RPC: Registered named UNIX socket transport module. [ 1.049178] RPC: Registered udp transport module. [ 1.051097] RPC: Registered tcp transport module. [ 1.053085] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.055739] NET: Registered protocol family 44 [ 1.057225] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.059149] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.061397] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.064044] PCI: CLS 0 bytes, default 64 [ 1.065484] Unpacking initramfs... [ 2.434763] debug: unmapping init [mem 0xffff9f85fcc54000-0xffff9f85fffbffff] [ 2.439040] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.441409] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.444555] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.955131] Initialise system trusted keyrings [ 2.956647] Key type blacklist registered [ 2.958564] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.967450] zbud: loaded [ 2.970334] *** VALIDATE nfs *** [ 2.971731] *** VALIDATE nfs4 *** [ 2.973395] pstore: using deflate compression [ 2.977331] Platform Keyring initialized [ 3.077715] NET: Registered protocol family 38 [ 3.079935] Key type asymmetric registered [ 3.082169] Asymmetric key parser 'x509' registered [ 3.083655] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.086713] io scheduler mq-deadline registered [ 3.088448] io scheduler kyber registered [ 3.090152] io scheduler bfq registered [ 3.092070] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.095362] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.098792] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.101678] ACPI: Power Button [PWRF] [ 3.107367] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.113678] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.125768] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.132167] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.145323] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.173073] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.200355] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.208703] Non-volatile memory driver v1.3 [ 3.210420] Linux agpgart interface v0.103 [ 3.249196] virtio_blk virtio1: [vda] 149784 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.252285] vda: detected capacity change from 0 to 76689408 [ 3.277479] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.280927] vdb: detected capacity change from 0 to 1073741824 [ 3.303695] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.306548] vdc: detected capacity change from 0 to 2621440000 [ 3.322680] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.325833] vdd: detected capacity change from 0 to 2621440000 [ 3.344173] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.347537] vde: detected capacity change from 0 to 4294967296 [ 3.367849] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.370829] vdf: detected capacity change from 0 to 4294967296 [ 3.396669] libphy: Fixed MDIO Bus: probed [ 3.421341] usbcore: registered new interface driver usbserial_generic [ 3.424759] usbserial: USB Serial support registered for generic [ 3.427188] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.431049] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.432230] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.434131] mousedev: PS/2 mouse device common for all mice [ 3.436431] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.438497] rtc_cmos 00:05: RTC can wake from S4 [ 3.442242] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.442592] rtc_cmos 00:05: registered as rtc0 [ 3.446578] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.449243] intel_pstate: CPU model not supported [ 3.451333] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.454499] hid: raw HID events driver (C) Jiri Kosina [ 3.456435] usbcore: registered new interface driver usbhid [ 3.458357] usbhid: USB HID core driver [ 3.459844] drop_monitor: Initializing network drop monitor service [ 3.462191] Initializing XFRM netlink socket [ 3.464239] NET: Registered protocol family 10 [ 3.467828] Segment Routing with IPv6 [ 3.469375] NET: Registered protocol family 17 [ 3.471392] mpls_gso: MPLS GSO support [ 3.476703] RAS: Correctable Errors collector initialized. [ 3.478675] AVX version of gcm_enc/dec engaged. [ 3.480470] AES CTR mode by8 optimization enabled [ 3.556965] sched_clock: Marking stable (3556947742, 0)->(4407551560, -850603818) [ 3.560199] registered taskstats version 1 [ 3.562591] Loading compiled-in X.509 certificates [ 3.564198] zswap: loaded using pool lzo/zbud [ 3.582687] Key type big_key registered [ 3.591917] Key type encrypted registered [ 3.593434] ima: No TPM chip found, activating TPM-bypass! [ 3.595411] ima: Allocated hash algorithm: sha1 [ 3.596788] ima: No architecture policies found [ 3.598198] evm: Initialising EVM extended attributes: [ 3.599345] evm: security.selinux [ 3.599993] evm: security.ima [ 3.600791] evm: security.capability [ 3.602179] evm: HMAC attrs: 0x1 [ 3.604784] rtc_cmos 00:05: setting system clock to 2026-09-06 03:08:58 UTC (1788664138) [ 3.609384] debug: unmapping init [mem 0xffffffff8a803000-0xffffffff8a9fffff] [ 3.612290] debug: unmapping init [mem 0xffffffff89582000-0xffffffff89858fff] [ 3.620065] Write protecting the kernel read-only data: 28672k [ 3.622876] debug: unmapping init [mem 0xffffffff87c03000-0xffffffff87dfffff] [ 3.625068] debug: unmapping init [mem 0xffffffff88514000-0xffffffff885fffff] [ 3.656382] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.661284] systemd[1]: Detected virtualization kvm. [ 3.662521] systemd[1]: Detected architecture x86-64. [ 3.663628] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.689155] systemd[1]: No hostname configured. [ 3.691225] systemd[1]: Set hostname to . [ 3.693829] random: systemd: uninitialized urandom read (16 bytes read) [ 3.695882] systemd[1]: Initializing machine ID from random generator. [ 3.739858] random: ln: uninitialized urandom read (6 bytes read) [ 3.818170] random: systemd: uninitialized urandom read (16 bytes read) [ 3.821209] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.825944] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ 3.829700] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Apply Kernel Variables... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.330620] device-mapper: uevent: version 1.0.3 [ 4.332769] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.952833] virtio_net virtio0 ens2: renamed from eth0 [ 4.968646] random: fast init done [ 5.008939] scsi host0: ata_piix [ 5.029638] scsi host1: ata_piix [ 5.030760] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.032220] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.774061] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.118884] random: crng init done [ 10.120327] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.478676] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.777396] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.049211] SELinux: Disabled at runtime. [ 12.114861] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.123805] systemd[1]: Detected virtualization kvm. [ 12.125670] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.609840] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.614181] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.620706] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.627566] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.631497] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.639525] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.645259] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Di[ 12.782316] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS rectory Watch. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.138425] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.478085] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.535674] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.712442] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.725421] EDAC sbridge: Ver: 1.1.2 [ 15.385995] Key type dns_resolver registered [ 15.690881] NFS: Registering the id_resolver key type [ 15.693053] Key type id_resolver registered [ 15.694771] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg122-server login: [ 42.441456] libcfs: loading out-of-tree module taints kernel. [ 42.466360] Key type ._llcrypt registered [ 42.468090] Key type .llcrypt registered [ 42.526312] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 51.660719] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 52.226793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.234953] alg: No test for adler32 (adler32-zlib) [ 53.262537] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 54.020416] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 55.879769] Key type lgssc registered [ 57.317797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 72.652830] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 111.113111] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 113.860143] hrtimer: interrupt took 4134819 ns [ 124.653722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 124.679285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 125.949965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 125.978365] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 126.131294] Lustre: lustre-MDT0000: new disk, initializing [ 126.267881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 126.291646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 130.912457] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 145.295449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 145.434515] Lustre: 6479:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 145.457440] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 145.463602] Lustre: Skipped 1 previous similar message [ 145.573598] Lustre: lustre-MDT0001: new disk, initializing [ 145.641923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 145.684366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 145.703233] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 150.890380] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 156.465944] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 166.926033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 167.303791] Lustre: lustre-OST0000: new disk, initializing [ 167.307625] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 167.324956] Lustre: 8416:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 167.409588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 170.282851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 170.296444] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 170.394288] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 174.020326] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.386311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 190.508111] Lustre: lustre-OST0001: new disk, initializing [ 190.510849] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 190.517958] Lustre: 9491:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 190.579121] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 197.967976] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.254259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 200.272126] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 200.338948] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 210.997470] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 217.844572] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 225.181963] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing check_logdir /tmp/testlogs/ [ 229.853745] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing yml_node [ 234.446313] Lustre: DEBUG MARKER: Client: 2.17.58.4 [ 237.122133] Lustre: DEBUG MARKER: MDS: 2.17.58.4 [ 239.899500] Lustre: DEBUG MARKER: OSS: 2.17.58.4 [ 241.536790] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 23:12:56 EDT 2026 [ 260.252506] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 261.970539] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 279.009797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 279.018152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 279.032265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 282.593305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 282.593623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 287.712663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 287.721933] Lustre: Skipped 6 previous similar messages [ 292.832653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 292.836431] Lustre: Skipped 3 previous similar messages [ 293.344388] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 293.647413] Lustre: server umount lustre-MDT0000 complete [ 297.172066] LustreError: 6470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788664432 with bad export cookie 5464104117324784879 [ 297.174901] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 297.190800] LustreError: 6470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 297.953387] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 297.955947] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 297.969244] Lustre: Skipped 3 previous similar messages [ 297.986461] Lustre: Skipped 1 previous similar message [ 303.074686] LustreError: 6486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 303.110337] LustreError: 6486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 303.732956] Lustre: server umount lustre-MDT0001 complete [ 307.346426] Lustre: server umount lustre-OST0000 complete [ 310.992402] Lustre: server umount lustre-OST0001 complete [ 317.596284] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 326.559856] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 364.547864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 364.759079] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 364.787046] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 364.885331] Lustre: lustre-MDT0000: new disk, initializing [ 365.040230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 365.071541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 369.839187] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 380.326681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 380.479487] Lustre: 15043:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 380.528570] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 380.537534] Lustre: Skipped 1 previous similar message [ 380.625863] Lustre: lustre-MDT0001: new disk, initializing [ 380.710702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 380.747874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 380.754302] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 385.620506] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 395.986460] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 402.880436] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 409.332546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 409.515075] Lustre: lustre-OST0000: new disk, initializing [ 409.519255] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 409.525879] Lustre: 17012:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 409.625128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 410.838676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 410.848116] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 410.943057] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 415.309785] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 424.673636] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 429.338763] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 429.564694] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 434.647552] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 434.826745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 441.834332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 441.843888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 441.847453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 441.857600] Lustre: Skipped 1 previous similar message [ 441.878275] Lustre: Skipped 3 previous similar messages [ 443.159578] Lustre: server umount lustre-OST0000 complete [ 446.945415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.964750] Lustre: Skipped 1 previous similar message [ 452.886122] Lustre: server umount lustre-MDT0000 complete [ 456.610967] LustreError: 15037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788664591 with bad export cookie 5464104117324786412 [ 456.616256] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 456.637672] LustreError: 15037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 456.913693] Lustre: server umount lustre-MDT0001 complete [ 462.502566] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 23:16:37 (1788664597) [ 480.672507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 481.031211] LustreError: 19886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 481.113721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 485.408593] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 486.370439] LustreError: 19887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.465098] LustreError: 19886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 493.296502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 497.050751] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 503.975541] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 508.795250] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 510.857462] Lustre: Failing over lustre-MDT0000 [ 510.944826] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.955691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 510.964922] Lustre: Skipped 5 previous similar messages [ 511.079537] Lustre: server umount lustre-MDT0000 complete [ 516.064956] LustreError: 19887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 516.100535] LustreError: 19887:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 528.352604] LustreError: 19882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.368418] LustreError: 19882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 530.918147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 531.174821] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 537.058451] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 537.065562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 537.084797] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 537.576970] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 548.871469] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 556.513603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.516488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 556.545585] Lustre: Skipped 2 previous similar messages [ 556.559158] Lustre: Skipped 1 previous similar message [ 559.478058] Lustre: server umount lustre-MDT0000 complete [ 561.634227] LustreError: 19886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 561.651230] LustreError: 19886:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 563.318089] LustreError: 19868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788664698 with bad export cookie 5464104117324787826 [ 563.335574] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 563.884671] Lustre: server umount lustre-MDT0001 complete [ 572.992766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 573.634718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 573.638508] Lustre: Skipped 2 previous similar messages [ 578.744574] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 589.170514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 595.011593] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 603.542132] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 609.569307] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 611.766768] Lustre: Failing over lustre-MDT0000 [ 612.088655] Lustre: server umount lustre-MDT0000 complete [ 614.882573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 614.886957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 614.892276] LustreError: Skipped 1 previous similar message [ 614.892946] LustreError: 23220:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 614.892953] LustreError: 23220:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 614.925014] Lustre: Skipped 1 previous similar message [ 630.362211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 630.485629] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 630.707342] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 634.791025] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 634.808036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 634.812066] Lustre: Skipped 1 previous similar message [ 634.826525] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 635.209832] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 644.914605] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 650.211967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 650.222768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 655.511302] Lustre: server umount lustre-MDT0000 complete [ 659.281232] LustreError: 23203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788664794 with bad export cookie 5464104117324788764 [ 659.285357] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 659.294516] LustreError: 23203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 659.611649] Lustre: server umount lustre-MDT0001 complete [ 668.210629] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 23:20:02 (1788664802) [ 685.531873] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 697.125130] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 749.593580] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 762.307205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 762.551724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 762.573253] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 762.634841] Lustre: lustre-MDT0000: new disk, initializing [ 762.690355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 762.694551] Lustre: Skipped 2 previous similar messages [ 762.709891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 768.136284] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 780.137521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 780.323446] Lustre: 29615:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 780.648556] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 785.774374] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 797.221206] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 803.377849] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 811.381526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 811.589803] Lustre: 31583:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 813.639241] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 813.731036] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 813.842148] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 813.855436] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 819.930894] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 831.377873] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 839.839380] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 859.905255] Lustre: server umount lustre-OST0000 complete [ 865.249891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 865.257521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 865.262908] Lustre: Skipped 3 previous similar messages [ 865.274925] Lustre: Skipped 2 previous similar messages [ 873.687171] Lustre: server umount lustre-MDT0000 complete [ 875.497338] LustreError: 29627:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 875.519445] LustreError: 29627:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 876.511472] Lustre: 3613:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788664995/real 1788664995] req@ffff9f85431d1f80 x1875550339850624/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788665011 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [ 877.678490] LustreError: 29608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788665012 with bad export cookie 5464104117324789170 [ 877.691250] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 877.697061] LustreError: 29608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 878.189841] Lustre: server umount lustre-MDT0001 complete [ 890.690339] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 893.536251] Key type lgssc unregistered [ 893.785883] LNet: 33558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.802421] LNetError: 33558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 893.818031] LNet: Removed LNI 192.168.201.122@tcp [ 894.664152] Key type .llcrypt unregistered [ 894.667181] Key type ._llcrypt unregistered [ 922.257694] Key type ._llcrypt registered [ 922.260503] Key type .llcrypt registered [ 922.371483] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 923.665830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 923.703808] alg: No test for adler32 (adler32-zlib) [ 924.738871] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 924.944107] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 926.655232] Key type lgssc registered [ 927.979659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 936.133200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 936.154553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 937.603788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 942.668786] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 947.176470] LustreError: 34971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 947.209045] LustreError: 34971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 952.313559] LustreError: 34972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 952.372950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 952.719147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 957.535962] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 966.695806] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 973.147624] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 981.028569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 981.544406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 988.589518] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 997.297929] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 998.054070] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 1016.239392] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 23:25:49 (1788665149) [ 1018.154696] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1020.841904] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1028.767960] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 23:26:02 (1788665162) [ 1039.123102] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 23:26:13 (1788665173) [ 1040.557291] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1051.509772] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1051.526676] Lustre: Skipped 2 previous similar messages [ 1053.783976] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 23:26:27 (1788665187) [ 1056.570632] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1056.577594] Lustre: Skipped 2 previous similar messages [ 1065.275580] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1065.281352] Lustre: Skipped 1 previous similar message [ 1082.439313] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1082.444156] Lustre: Skipped 5 previous similar messages [ 1084.996679] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 23:26:59 (1788665219) [ 1088.050625] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1091.964938] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1102.105823] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 23:27:16 (1788665236) [ 1116.223852] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 1116.230339] Lustre: Skipped 7 previous similar messages [ 1118.185458] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 23:27:32 (1788665252) [ 1136.098148] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1136.116379] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1136.143131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1136.609248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1136.623915] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1136.654051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1141.733786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1146.854082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1146.857532] Lustre: Skipped 1 previous similar message [ 1147.871133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1148.090371] Lustre: server umount lustre-OST0000 complete [ 1151.971237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1151.979880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1151.980153] Lustre: Skipped 1 previous similar message [ 1151.986626] Lustre: Skipped 2 previous similar messages [ 1162.210150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1162.224660] Lustre: Skipped 3 previous similar messages [ 1166.303880] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1166.456939] Lustre: server umount lustre-MDT0000 complete [ 1167.357387] LustreError: 34967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1167.372850] LustreError: 34967:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1170.061755] LustreError: 34952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788665304 with bad export cookie 4771562381561991766 [ 1170.062297] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1170.070402] LustreError: 34952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1170.669639] Lustre: server umount lustre-MDT0001 complete [ 1183.165464] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 1186.152423] Key type lgssc unregistered [ 1186.633432] LNet: 42839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1186.652301] LNetError: 42839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1186.690811] LNet: Removed LNI 192.168.201.122@tcp [ 1187.831951] Key type .llcrypt unregistered [ 1187.833741] Key type ._llcrypt unregistered [ 1210.209755] Key type ._llcrypt registered [ 1210.211859] Key type .llcrypt registered [ 1210.357799] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 1229.204614] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 1230.532438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1230.569515] alg: No test for adler32 (adler32-zlib) [ 1231.638899] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 1231.850104] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 1233.567284] Key type lgssc registered [ 1234.989131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1273.811779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1273.839973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1275.163381] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1275.190309] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1275.288459] Lustre: lustre-MDT0000: new disk, initializing [ 1275.381280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1275.420351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1280.029379] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1292.970765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1293.179966] Lustre: 45991:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1293.265230] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1293.269467] Lustre: Skipped 1 previous similar message [ 1293.353311] Lustre: lustre-MDT0001: new disk, initializing [ 1293.478645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1293.545476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1293.571992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1300.358716] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1311.694825] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1319.474968] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1327.644674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1327.938878] Lustre: lustre-OST0000: new disk, initializing [ 1327.946291] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1327.954677] Lustre: 47961:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1328.048833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1336.118508] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1336.415155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1336.434304] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1336.529211] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1345.919730] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1350.927523] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1351.199469] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1356.154870] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1356.360440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1362.400407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1362.400780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1362.415957] LustreError: Skipped 1 previous similar message [ 1362.421194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1362.437343] Lustre: Skipped 1 previous similar message [ 1362.471857] Lustre: Skipped 1 previous similar message [ 1364.620681] Lustre: server umount lustre-OST0000 complete [ 1370.593697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1370.597254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1370.625971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1372.643578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1372.644190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1374.744583] Lustre: server umount lustre-MDT0000 complete [ 1378.682510] LustreError: 45982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788665513 with bad export cookie 18312686476677948698 [ 1378.698735] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1379.270374] Lustre: server umount lustre-MDT0001 complete [ 1390.751081] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 23:32:05 (1788665525) [ 1397.479486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1397.877603] LustreError: 50423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1397.906064] LustreError: 50423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1397.977737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1403.063139] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1403.364052] LustreError: 50424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1407.463589] LustreError: 50423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1412.282542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1412.634316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1417.813635] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1426.875726] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1434.426100] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1442.200878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1442.284783] LustreError: 52244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1442.624077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1449.572971] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1458.484812] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1464.218862] LustreError: 52953:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1465.088260] Lustre: Modifying parameter general.timeout=1 in log params [ 1466.875981] Lustre: Modifying parameter general.timeout=3 in log params [ 1466.880558] Lustre: Skipped 1 previous similar message [ 1468.941611] Lustre: Modifying parameter general.timeout=5 in log params [ 1468.944949] Lustre: Skipped 1 previous similar message [ 1473.681148] Lustre: Modifying parameter general.timeout=10 in log params [ 1473.686516] Lustre: Skipped 4 previous similar messages [ 1481.815794] Lustre: Modifying parameter general.timeout=19 in log params [ 1481.820765] Lustre: Skipped 8 previous similar messages [ 1498.605605] Lustre: Modifying parameter general.timeout=35 in log params [ 1498.608152] Lustre: Skipped 15 previous similar messages [ 1531.331962] Lustre: Modifying parameter general.timeout=65 in log params [ 1531.340403] Lustre: Skipped 29 previous similar messages [ 1604.245772] Lustre: Disabling parameter general.timeout= in log params [ 1604.249737] Lustre: Skipped 64 previous similar messages [ 1608.753505] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 23:35:42 (1788665742) [ 1612.078357] Lustre: 59906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1614.505907] Lustre: 60004:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1616.773157] Lustre: 60117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1622.807778] Lustre: 60415:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1622.816252] Lustre: 60415:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1649.304775] Lustre: 61787:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1649.345318] LustreError: 61787:0:(mgs_handler.c:1205:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1653.937403] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 23:36:28 (1788665788) [ 1668.581574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1668.599084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1668.628233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1668.631046] Lustre: Skipped 1 previous similar message [ 1672.982978] Lustre: server umount lustre-MDT0000 complete [ 1675.746373] LustreError: 50423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1675.779306] LustreError: 50423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1677.112364] LustreError: 53201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788665812 with bad export cookie 18312686476677949573 [ 1677.117985] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1677.122491] LustreError: 53201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 1677.596383] Lustre: server umount lustre-MDT0001 complete [ 1693.072470] Lustre: server umount lustre-OST0000 complete [ 1723.312521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1723.650831] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1723.933033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1728.848095] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1738.296633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1738.363391] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1738.380919] Lustre: 64317:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1744.288772] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1753.031279] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1760.021534] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1768.667108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1768.942668] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1769.321286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1769.328785] Lustre: Skipped 1 previous similar message [ 1776.792093] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1784.978144] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1786.341182] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1786.347634] Lustre: Skipped 23 previous similar messages [ 1786.384801] Lustre: 66886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1793.505554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1793.520538] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1793.536765] Lustre: Skipped 2 previous similar messages [ 1793.554531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1793.561082] Lustre: Skipped 3 previous similar messages [ 1799.164587] LustreError: 66167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1799.298499] Lustre: server umount lustre-OST0000 complete [ 1804.267560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1804.272852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1804.286118] Lustre: Skipped 2 previous similar messages [ 1804.289804] Lustre: Skipped 2 previous similar messages [ 1818.079533] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1818.328423] Lustre: server umount lustre-MDT0000 complete [ 1819.628384] LustreError: 64331:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1819.656421] LustreError: 64331:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1823.090534] LustreError: 64310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788665957 with bad export cookie 18312686476677952779 [ 1823.102644] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1823.109672] LustreError: 64310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1823.532400] Lustre: server umount lustre-MDT0001 complete [ 1837.806985] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 1840.870683] Key type lgssc unregistered [ 1841.382634] LNet: 68096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1841.392557] LNetError: 68096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1841.441720] LNet: Removed LNI 192.168.201.122@tcp [ 1842.517327] Key type .llcrypt unregistered [ 1842.525087] Key type ._llcrypt unregistered [ 1870.476491] Key type ._llcrypt registered [ 1870.478311] Key type .llcrypt registered [ 1870.608530] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 1871.891314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1871.952675] alg: No test for adler32 (adler32-zlib) [ 1873.332684] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 1873.528528] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 1875.183534] Key type lgssc registered [ 1876.628545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1885.821781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1885.863788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1887.613898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1893.052699] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1897.959916] LustreError: 69509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1897.972922] LustreError: 69509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1903.072529] LustreError: 69508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1903.309497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1904.140567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1908.625740] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1916.471983] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1923.188699] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1931.684466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1932.225548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1938.737289] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1946.586576] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1955.548293] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1965.024489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1965.031748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1965.048320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1969.121370] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1969.123928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1970.425441] Lustre: server umount lustre-MDT0000 complete [ 1974.166780] LustreError: 72194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666109 with bad export cookie 14306153529281617973 [ 1974.168748] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1974.180280] LustreError: 72194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1974.240476] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1974.242436] LustreError: 69504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1974.245799] Lustre: Skipped 1 previous similar message [ 1974.265742] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1974.277063] Lustre: Skipped 2 previous similar messages [ 1974.425810] Lustre: server umount lustre-MDT0001 complete [ 1980.649893] Lustre: server umount lustre-OST0000 complete [ 2009.686691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2010.072752] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2010.138743] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2010.494927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2015.636447] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2027.236339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2027.409238] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2027.427673] Lustre: 74378:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2028.141069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2033.550443] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2043.112690] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2050.173179] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2058.002519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2058.257102] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2058.698799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2065.570661] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2073.719220] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2076.011889] Lustre: 76955:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2078.183872] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2085.349689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2085.361523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2085.372030] Lustre: Skipped 1 previous similar message [ 2085.393626] Lustre: Skipped 1 previous similar message [ 2090.475718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2090.493875] Lustre: Skipped 1 previous similar message [ 2091.381624] Lustre: server umount lustre-OST0000 complete [ 2095.587924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2095.614928] Lustre: Skipped 1 previous similar message [ 2100.704270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2100.716863] Lustre: Skipped 3 previous similar messages [ 2101.610023] Lustre: server umount lustre-MDT0000 complete [ 2105.832721] LustreError: 74387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2105.859388] LustreError: 74387:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2106.101427] LustreError: 76246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666240 with bad export cookie 14306153529281618806 [ 2106.115692] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2106.116783] LustreError: 76246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2106.572746] Lustre: server umount lustre-MDT0001 complete [ 2120.121817] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 2122.675197] Key type lgssc unregistered [ 2123.044457] LNet: 78166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2123.050278] LNetError: 78166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2123.063767] LNet: Removed LNI 192.168.201.122@tcp [ 2124.101175] Key type .llcrypt unregistered [ 2124.103809] Key type ._llcrypt unregistered [ 2151.292507] Key type ._llcrypt registered [ 2151.294955] Key type .llcrypt registered [ 2151.509801] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 2152.999896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2153.262460] alg: No test for adler32 (adler32-zlib) [ 2154.560207] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 2154.883969] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 2156.600863] Key type lgssc registered [ 2158.396392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2168.755169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2168.776197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2170.292249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2176.002904] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2180.595796] LustreError: 79580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2180.624168] LustreError: 79580:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2185.707057] LustreError: 79581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2185.945888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2186.465975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2191.004645] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2199.433913] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2206.169848] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2214.417728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2215.015241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2222.087258] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2232.095067] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2238.797379] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2240.026857] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2241.733848] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2241.738678] Lustre: Skipped 1 previous similar message [ 2244.686446] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2244.694419] Lustre: Skipped 2 previous similar messages [ 2249.087110] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2249.094278] Lustre: Skipped 3 previous similar messages [ 2257.207961] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2257.211583] Lustre: Skipped 7 previous similar messages [ 2273.905318] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2273.909316] Lustre: Skipped 17 previous similar messages [ 2305.920882] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2305.923395] Lustre: Skipped 30 previous similar messages [ 2345.247832] Lustre: 87688:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2347.969658] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 23:48:02 (1788666482) [ 2349.502919] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2351.260041] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 23:48:05 (1788666485) [ 2359.780217] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2359.791714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2359.795233] Lustre: Skipped 1 previous similar message [ 2359.800929] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2361.725462] Lustre: server umount lustre-OST0000 complete [ 2365.418559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2365.439836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2365.457394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2365.461841] Lustre: Skipped 1 previous similar message [ 2370.018709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2370.023867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2371.718738] Lustre: server umount lustre-MDT0000 complete [ 2375.139455] LustreError: 87501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2375.170171] LustreError: 87501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2375.555936] LustreError: 79561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666510 with bad export cookie 17689959243042088555 [ 2375.556624] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2375.581641] LustreError: 79561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2375.952532] Lustre: server umount lustre-MDT0001 complete [ 2388.537786] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 2391.419220] Key type lgssc unregistered [ 2391.663791] LNet: 88936:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2391.669391] LNetError: 88936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2392.680857] LNet: Removed LNI 192.168.201.122@tcp [ 2393.670593] Key type .llcrypt unregistered [ 2393.676352] Key type ._llcrypt unregistered [ 2398.620560] Key type ._llcrypt registered [ 2398.626319] Key type .llcrypt registered [ 2398.849741] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2404.082413] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules [ 2405.458823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2405.523702] alg: No test for adler32 (adler32-zlib) [ 2406.549356] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 2406.810369] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 2408.487329] Key type lgssc registered [ 2409.761520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2410.856334] LustreError: 90134:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2429.928399] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 2433.100963] LustreError: 90134:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2439.737118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2439.776815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2441.652934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2447.599594] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2451.942675] LustreError: 91133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2451.984891] LustreError: 91133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2457.061521] LustreError: 91134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2460.347939] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 23:49:54 (1788666594) [ 2462.181897] LustreError: 91133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2466.280746] LustreError: 91134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2466.915027] LustreError: 91165:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2466.932824] LustreError: 91165:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2466.952431] LustreError: 91165:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 25, retries 0, failed: rc = -5 [ 2467.411303] Lustre: server umount lustre-MDT0000 complete [ 2484.156699] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 2487.890617] Key type lgssc unregistered [ 2488.282338] LNet: 92658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2488.294785] LNetError: 92658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2488.327435] LNet: Removed LNI 192.168.201.122@tcp [ 2489.543856] Key type .llcrypt unregistered [ 2489.546935] Key type ._llcrypt unregistered [ 2509.376082] Key type ._llcrypt registered [ 2509.383421] Key type .llcrypt registered [ 2509.501715] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 2510.549085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2510.718235] alg: No test for adler32 (adler32-zlib) [ 2511.770660] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 2512.033074] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 2513.720288] Key type lgssc registered [ 2514.769310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2522.806538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2522.846822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2524.643172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2530.069391] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2534.882021] LustreError: 93878:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2534.903931] LustreError: 93878:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2539.997730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2540.014019] LustreError: 93877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2540.696202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2545.513989] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2553.907202] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2559.863702] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2567.345543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2567.867361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2574.275121] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2582.374083] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2632.087202] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 23:52:46 (1788666766) [ 2633.860792] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2635.865987] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 23:52:50 (1788666770) [ 2641.893733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2641.909237] Lustre: Skipped 2 previous similar messages [ 2641.909878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2641.927076] Lustre: Skipped 2 previous similar messages [ 2646.986359] Lustre: server umount lustre-MDT0000 complete [ 2647.017100] LustreError: 93878:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2647.049747] LustreError: 93878:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2651.090743] LustreError: 93857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666785 with bad export cookie 11172556020369820547 [ 2651.105215] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2651.425196] Lustre: server umount lustre-MDT0001 complete [ 2665.561422] Lustre: server umount lustre-OST0000 complete [ 2678.834958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2679.365787] LustreError: 98152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2679.611657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2684.388972] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2684.905676] LustreError: 98153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2692.999585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2693.756950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2698.955841] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2707.129502] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2713.536865] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2728.128891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2728.287909] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2728.293606] LustreError: 98142:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2728.313862] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2728.321558] LustreError: 100238:0:(tgt_mount.c:2619:server_fill_super()) Unable to start targets: -98 [ 2728.326767] LustreError: 100238:0:(tgt_mount.c:2134:server_put_super()) no obd lustre-OST0000 [ 2728.331312] LustreError: 100238:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2728.384276] Lustre: server umount lustre-OST0000 complete [ 2728.388275] LustreError: 100238:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2737.891564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2738.129406] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2738.148210] LustreError: 98142:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2738.208308] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2738.221166] LustreError: 100702:0:(tgt_mount.c:2619:server_fill_super()) Unable to start targets: -98 [ 2738.231255] LustreError: 100702:0:(tgt_mount.c:2134:server_put_super()) no obd lustre-OST0000 [ 2738.263248] LustreError: 100702:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2738.387798] Lustre: server umount lustre-OST0000 complete [ 2738.390310] LustreError: 100702:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2748.343054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2748.566901] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2748.581477] Lustre: Found index 0 for lustre-OST0000, updating log [ 2748.590837] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2748.703938] Lustre: lustre-OST0000: new disk, initializing [ 2748.709597] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2748.726057] Lustre: 101237:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2748.973079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2755.419728] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2759.831213] Lustre: Failing over lustre-OST0000 [ 2760.137738] Lustre: server umount lustre-OST0000 complete [ 2765.285392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2765.290558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2765.302088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2765.315744] Lustre: Skipped 1 previous similar message [ 2766.373385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2766.384322] Lustre: Skipped 1 previous similar message [ 2771.424784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2771.437135] Lustre: Skipped 1 previous similar message [ 2774.670792] Lustre: server umount lustre-MDT0000 complete [ 2776.545725] LustreError: 98149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2776.576766] LustreError: 98149:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2778.848511] LustreError: 98894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666913 with bad export cookie 11172556020369823291 [ 2778.859756] LustreError: 98894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2778.862033] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2779.397842] Lustre: server umount lustre-MDT0001 complete [ 2790.458823] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 23:55:24 (1788666924) [ 2809.993799] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 2820.271827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2820.721760] LustreError: 103584:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2820.838707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2826.097914] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2836.499198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2842.387368] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2845.664169] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2853.371416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2854.264813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2854.271718] Lustre: Skipped 1 previous similar message [ 2854.284593] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2861.425800] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2861.439045] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2861.475135] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2863.241541] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2865.583572] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:05 [ 2865.608540] Lustre: lustre-OST0000: Recovery over after 0:04, of 1 clients 1 recovered and 0 were evicted. [ 2865.612274] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2865.630820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2865.631200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2873.377454] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2873.529103] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2873.678487] Lustre: lustre-OST0001: new disk, initializing [ 2873.688820] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2873.705637] Lustre: 105909:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2880.857480] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2881.667412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2881.692836] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2881.815185] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2890.203367] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2896.583090] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2902.504975] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2902.507837] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2902.531143] Lustre: Skipped 3 previous similar messages [ 2902.555948] Lustre: Skipped 2 previous similar messages [ 2912.736855] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2912.749174] Lustre: Skipped 7 previous similar messages [ 2915.295441] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2915.565028] Lustre: server umount lustre-MDT0001 complete [ 2917.865399] LustreError: 103584:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2917.893714] LustreError: 103584:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2927.067099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2927.207661] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2927.212528] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2927.303868] Lustre: 103571:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2928.289740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2928.304928] Lustre: Skipped 1 previous similar message [ 2928.533525] LustreError: 107426:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2928.556836] LustreError: 107426:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2928.580705] LustreError: 103576:0:(mgc_request.c:612:do_requeue()) failed processing log: -17 [ 2928.757739] LustreError: 107441:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.122@tcp: cfg command failed: rc = -17 [ 2928.769488] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2928.769488] [ 2933.599891] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2933.750706] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2945.827425] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 23:58:00 (1788667080) [ 2969.161838] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 2989.946381] Lustre: 109543:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3017.141907] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3020.766480] Lustre: 110678:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3117.538780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3117.544747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3117.556838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3117.566128] Lustre: Skipped 1 previous similar message [ 3119.931114] Lustre: server umount lustre-MDT0000 complete [ 3123.168175] LustreError: 103579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3123.191863] LustreError: 103579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3124.003232] LustreError: 103564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667258 with bad export cookie 11172556020369824292 [ 3124.005343] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3124.020553] LustreError: 103564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3124.309596] Lustre: server umount lustre-MDT0001 complete [ 3138.549057] Lustre: server umount lustre-OST0000 complete [ 3153.455881] Lustre: server umount lustre-OST0001 complete [ 3160.155857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3171.143354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3181.173567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3191.944423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3216.346077] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 3228.827762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3228.844495] Lustre: lustre-MDT0000: reset Object Index mappings [ 3229.449139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3234.707180] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3243.860654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3243.882557] Lustre: lustre-MDT0001: reset Object Index mappings [ 3248.993759] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3251.874958] Lustre: 115795:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3259.263896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3267.081446] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3269.357332] LustreError: 116149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3269.374299] LustreError: 116149:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3269.399778] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3274.496541] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:386 to 0x280000401:418) [ 3275.789493] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3281.924622] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3281.940778] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3283.627565] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3292.545225] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3296.784933] Lustre: 117674:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3352.306746] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 00:04:46 (1788667486) [ 3377.634565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3377.644902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3377.655489] Lustre: Skipped 3 previous similar messages [ 3377.666143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3377.674403] Lustre: Skipped 4 previous similar messages [ 3382.382449] Lustre: server umount lustre-MDT0000 complete [ 3386.215634] LustreError: 116983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667521 with bad export cookie 11172556020369871346 [ 3386.226540] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3389.409295] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3389.426986] Lustre: Skipped 4 previous similar messages [ 3392.775295] Lustre: server umount lustre-MDT0001 complete [ 3402.787422] Lustre: server umount lustre-OST0000 complete [ 3412.868957] Lustre: server umount lustre-OST0001 complete [ 3421.201484] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 3437.393673] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 3477.013304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3477.402898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3477.432085] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3477.505983] Lustre: lustre-MDT0000: new disk, initializing [ 3477.591204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3483.201961] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3493.113492] Lustre: server umount lustre-MDT0000 complete [ 3504.927970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3505.359034] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3505.375311] Lustre: 121964:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3505.398954] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3505.653354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3505.667308] Lustre: Skipped 4 previous similar messages [ 3511.368418] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3516.877232] Lustre: server umount lustre-MDT0000 complete [ 3526.207341] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 00:07:40 (1788667660) [ 3528.103648] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3530.308966] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 00:07:44 (1788667664) [ 3548.009413] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 3558.349911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3563.002955] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3571.611143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3571.700257] Lustre: 124015:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3571.752315] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3571.755610] Lustre: Skipped 1 previous similar message [ 3571.809330] Lustre: lustre-MDT0001: new disk, initializing [ 3571.920648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3571.935319] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3575.856360] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3581.445244] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3588.229840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3588.519374] Lustre: lustre-OST0000: new disk, initializing [ 3588.523436] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3588.535439] Lustre: 125591:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3590.293705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3590.304571] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3590.459320] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3595.887466] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3608.793684] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3609.072838] Lustre: lustre-OST0001: new disk, initializing [ 3609.076414] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3609.082925] Lustre: 126462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3610.999544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3611.005317] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3611.105901] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3617.800886] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3628.767485] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3633.026742] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4037.036574] Lustre: 132884:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4037.040627] Lustre: 132884:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4037.044656] Lustre: 132884:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4037.051693] Lustre: 132884:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4037.054637] Lustre: 132884:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4037.057820] Lustre: 132884:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4311.764810] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:20:45 (1788668445) [ 4329.953236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4329.962233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4329.978083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4329.983497] Lustre: Skipped 8 previous similar messages [ 4333.613990] Lustre: server umount lustre-MDT0000 complete [ 4336.105729] LustreError: 127286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4336.138149] LustreError: 127286:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 4337.408497] LustreError: 132939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668472 with bad export cookie 11172556020369939617 [ 4337.410289] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4337.430909] LustreError: 132939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4341.221603] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4341.224594] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4341.238508] Lustre: Skipped 4 previous similar messages [ 4341.250859] Lustre: Skipped 5 previous similar messages [ 4343.920273] Lustre: server umount lustre-MDT0001 complete [ 4353.810893] Lustre: server umount lustre-OST0000 complete [ 4364.070749] Lustre: server umount lustre-OST0001 complete [ 4372.141661] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 4385.232498] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 4425.837769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4426.150233] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4426.194974] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4426.276473] Lustre: lustre-MDT0000: new disk, initializing [ 4426.361595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4426.366259] Lustre: Skipped 4 previous similar messages [ 4426.377934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4432.477227] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4445.717779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4445.846048] Lustre: 135937:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4445.940070] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4445.944386] Lustre: Skipped 1 previous similar message [ 4446.045878] Lustre: lustre-MDT0001: new disk, initializing [ 4446.203557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4446.208916] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4452.100901] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4464.080894] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4471.124630] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4480.129417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4480.546607] Lustre: lustre-OST0000: new disk, initializing [ 4480.550062] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4480.554991] Lustre: 137906:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4481.696667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4481.704299] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4481.801780] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4486.977187] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4497.580128] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4509.079670] Lustre: lustre-MDD0000: changelog on [ 4512.375804] Lustre: lustre-MDD0001: changelog on [ 6488.565259] Lustre: lustre-MDD0001: changelog off [ 6490.609571] Lustre: 135942:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6491.740282] Lustre: lustre-MDD0000: changelog off [ 6499.303832] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6499.305188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6499.312134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6499.312153] Lustre: Skipped 2 previous similar messages [ 6499.336102] Lustre: Skipped 1 previous similar message [ 6504.427877] LustreError: 139539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6504.453364] LustreError: 139539:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6504.570594] Lustre: server umount lustre-OST0000 complete [ 6509.536432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6509.541195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6509.549040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6509.549047] Lustre: Skipped 1 previous similar message [ 6509.571832] Lustre: Skipped 1 previous similar message [ 6514.656627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6514.663617] Lustre: Skipped 2 previous similar messages [ 6515.912992] Lustre: server umount lustre-MDT0000 complete [ 6519.776206] LustreError: 137175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6519.795173] LustreError: 137175:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6520.309631] LustreError: 135929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670655 with bad export cookie 11172556020370719662 [ 6520.310610] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6520.323568] LustreError: 135929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6520.593441] Lustre: server umount lustre-MDT0001 complete [ 6531.945277] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 6534.645381] Key type lgssc unregistered [ 6534.995101] LNet: 147071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6535.001417] LNetError: 147071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6535.033489] LNet: Removed LNI 192.168.201.122@tcp [ 6535.898455] Key type .llcrypt unregistered [ 6535.901747] Key type ._llcrypt unregistered [ 6555.301775] Key type ._llcrypt registered [ 6555.310693] Key type .llcrypt registered [ 6555.431196] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 6570.561326] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 6572.079984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6572.171279] alg: No test for adler32 (adler32-zlib) [ 6573.412744] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 6573.694974] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 6575.364688] Key type lgssc registered [ 6576.741496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6613.752326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6613.773714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6615.004796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6615.025661] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6615.097747] Lustre: lustre-MDT0000: new disk, initializing [ 6615.173110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6615.191056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6619.650301] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6630.434076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6630.595869] Lustre: 150269:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6630.706580] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6630.718123] Lustre: Skipped 1 previous similar message [ 6630.808205] Lustre: lustre-MDT0001: new disk, initializing [ 6630.947729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6631.000020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6631.022491] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6636.122893] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6646.086742] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6652.254812] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6661.340189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6661.819144] Lustre: lustre-OST0000: new disk, initializing [ 6661.823276] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6661.828481] Lustre: 152235:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6661.987098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6669.171843] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6671.467423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6671.484092] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6671.657816] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6680.629076] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6688.316725] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_module obdecho/obdecho [ 6690.377243] LustreError: 153240:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6690.386267] LustreError: 153240:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6691.991673] LustreError: 153337:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6707.681275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6707.682824] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6707.698384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6707.720196] Lustre: Skipped 1 previous similar message [ 6707.747624] Lustre: Skipped 2 previous similar messages [ 6712.814493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6712.819437] Lustre: Skipped 2 previous similar messages [ 6713.464268] Lustre: server umount lustre-MDT0000 complete [ 6717.959113] LustreError: 150259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670852 with bad export cookie 12058684640404968950 [ 6717.964155] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6717.971233] LustreError: 150259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6718.426292] Lustre: server umount lustre-MDT0001 complete [ 6733.127600] Lustre: server umount lustre-OST0000 complete [ 6762.770650] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 6777.919161] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 6821.084613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6821.349979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6821.382425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6821.476943] Lustre: lustre-MDT0000: new disk, initializing [ 6821.579087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6821.609411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6826.057795] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6837.764160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6837.913736] Lustre: 157095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6837.976250] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6837.982270] Lustre: Skipped 1 previous similar message [ 6838.113994] Lustre: lustre-MDT0001: new disk, initializing [ 6838.254701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6838.297545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6838.306650] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6842.779550] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6851.653792] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6857.022351] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6863.553799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6863.786317] Lustre: lustre-OST0000: new disk, initializing [ 6863.789942] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6863.794643] Lustre: 159061:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6863.882430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6869.502974] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6873.147162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6873.162410] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6873.356530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6879.189248] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6885.036226] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6885.401932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6890.457255] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6890.683258] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6894.049744] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6894.052190] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6894.060729] LustreError: Skipped 1 previous similar message [ 6894.061783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6894.078501] Lustre: Skipped 2 previous similar messages [ 6898.713205] Lustre: server umount lustre-OST0000 complete [ 6904.293519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6904.316182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6904.321140] Lustre: Skipped 2 previous similar messages [ 6908.513626] Lustre: server umount lustre-MDT0000 complete [ 6909.409688] LustreError: 160401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6909.438775] LustreError: 160401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6912.222834] LustreError: 157087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671047 with bad export cookie 12058684640404969909 [ 6912.232762] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6912.622730] Lustre: server umount lustre-MDT0001 complete [ 6916.388990] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 01:04:10 (1788671050) [ 6918.290828] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6920.400907] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 01:04:14 (1788671054) [ 6928.817562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6929.118014] LustreError: 161473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6929.181723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6934.104772] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6934.499884] LustreError: 161474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6939.617225] LustreError: 161473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6942.806921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6947.903477] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6955.471386] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6961.157393] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6967.899181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6968.232433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6968.242329] Lustre: Skipped 1 previous similar message [ 6974.221935] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6981.777839] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6989.283223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6989.290419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6989.305533] Lustre: Skipped 1 previous similar message [ 6989.309370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6994.209957] Lustre: server umount lustre-MDT0000 complete [ 6994.921273] LustreError: 161469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6994.955920] LustreError: 161469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6998.480961] LustreError: 161455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671133 with bad export cookie 12058684640404970791 [ 6998.484491] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6998.504180] LustreError: 161455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7000.033821] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7000.048980] Lustre: Skipped 2 previous similar messages [ 7000.059623] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7000.062431] Lustre: Skipped 3 previous similar messages [ 7003.011550] LustreError: 162713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7003.039312] LustreError: 162713:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7004.821931] Lustre: server umount lustre-MDT0001 complete [ 7011.194158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7024.503924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7024.698564] LustreError: 165314:0:(llog.c:1655:llog_backup()) MGC192.168.201.122@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7024.706754] Lustre: 165314:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.122@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7028.709777] LustreError: 165319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7064.444773] LustreError: 165321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7064.492930] LustreError: 165321:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [ 7085.538320] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7085.544447] Lustre: 165314:0:(mgc_request.c:1898:mgc_process_log()) MGC192.168.201.122@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7126.533520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7129.983146] LustreError: 165339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7130.006124] LustreError: 165339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 43 previous similar messages [ 7131.505139] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7131.620591] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7139.786938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7143.905053] LustreError: 148490:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000a9bdeb4 x1875557176440064/t0(0) o250->MGC192.168.201.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7148.863645] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7149.548324] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7157.111908] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7159.353593] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7165.694343] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7167.616480] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7185.398396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7185.402206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7185.412780] Lustre: Skipped 2 previous similar messages [ 7185.416137] Lustre: Skipped 3 previous similar messages [ 7185.425084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7191.429896] Lustre: server umount lustre-MDT0000 complete [ 7195.349912] LustreError: 161455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671330 with bad export cookie 12058684640404977245 [ 7195.354612] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7195.366393] LustreError: 161455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7195.599415] Lustre: server umount lustre-MDT0001 complete [ 7209.776902] Lustre: server umount lustre-OST0000 complete [ 7231.289792] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 7243.197203] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 7275.979112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7276.318431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7276.388169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7276.496397] Lustre: lustre-MDT0000: new disk, initializing [ 7276.572298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7276.578887] Lustre: Skipped 1 previous similar message [ 7276.589280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7280.208798] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7290.584694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7290.676620] Lustre: 170419:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7291.091784] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7296.117456] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7307.508522] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7313.279928] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7320.716917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7321.069753] Lustre: lustre-OST0000: new disk, initializing [ 7321.071937] Lustre: Skipped 1 previous similar message [ 7321.078128] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7321.084176] Lustre: Skipped 2 previous similar messages [ 7321.094247] Lustre: 172390:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7326.888898] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7330.901537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7330.915299] Lustre: Skipped 1 previous similar message [ 7330.922566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7331.018250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7335.880049] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7340.625273] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7340.829535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7345.129138] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7345.283813] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7351.794277] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7351.799350] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7351.800834] LustreError: Skipped 1 previous similar message [ 7351.802459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7351.802466] Lustre: Skipped 4 previous similar messages [ 7351.848786] Lustre: Skipped 1 previous similar message [ 7353.361739] Lustre: server umount lustre-OST0000 complete [ 7363.978081] Lustre: server umount lustre-MDT0000 complete [ 7367.136578] LustreError: 170432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7367.157479] LustreError: 170432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 7368.325801] LustreError: 171215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671503 with bad export cookie 12058684640404994080 [ 7368.335230] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7368.337824] LustreError: 171215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7368.650612] Lustre: server umount lustre-MDT0001 complete [ 7378.030661] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 01:11:52 (1788671512) [ 7385.106799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7393.033402] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7403.195607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7408.425433] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7415.921762] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7424.018472] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7431.257922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7437.365145] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7444.552782] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7468.755361] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 01:13:23 (1788671603) [ 7475.680109] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7475.694221] LustreError: Skipped 1 previous similar message [ 7475.699353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7475.717799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7475.733161] Lustre: Skipped 3 previous similar messages [ 7475.750314] Lustre: Skipped 5 previous similar messages [ 7481.166264] Lustre: server umount lustre-OST0000 complete [ 7492.916949] Lustre: server umount lustre-MDT0000 complete [ 7497.183266] LustreError: 174782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671632 with bad export cookie 12058684640404994962 [ 7497.184896] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7497.196895] LustreError: 174782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7497.642576] Lustre: server umount lustre-MDT0001 complete [ 7509.276745] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 7511.957411] Key type lgssc unregistered [ 7512.235720] LNet: 178553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7512.246814] LNetError: 178553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7512.265257] LNet: Removed LNI 192.168.201.122@tcp [ 7513.081558] Key type .llcrypt unregistered [ 7513.086680] Key type ._llcrypt unregistered [ 7531.634706] Key type ._llcrypt registered [ 7531.640232] Key type .llcrypt registered [ 7531.812412] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 7532.894470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7533.095555] alg: No test for adler32 (adler32-zlib) [ 7534.108284] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 7534.295144] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 7535.951312] Key type lgssc registered [ 7537.360918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7545.019318] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7545.050659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7560.607891] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7565.727425] LustreError: 179794:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.122@tcp: failed processing log, type 4: rc = -110 [ 7570.849557] LustreError: 179794:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7570.863368] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7570.877509] LustreError: MGC192.168.201.122@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7570.900956] LustreError: 179794:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7570.919980] LustreError: 179794:0:(tgt_mount.c:2619:server_fill_super()) Unable to start targets: -22 [ 7570.928861] LustreError: 179794:0:(tgt_mount.c:2134:server_put_super()) no obd lustre-OST0000 [ 7571.000507] Lustre: server umount lustre-OST0000 complete [ 7571.003943] LustreError: 179794:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7581.245094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7582.794955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7587.025675] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7587.813685] LustreError: 180293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7587.842151] LustreError: 180293:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7592.929933] LustreError: 180294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7595.442882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7596.050151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7601.070597] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7609.308386] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7614.543315] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7621.717611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7621.966334] LustreError: 182110:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7621.982402] Lustre: 182110:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.122@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7622.233210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7628.986091] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7637.337828] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7640.870201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7640.879734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7640.905201] Lustre: Skipped 2 previous similar messages [ 7640.926645] Lustre: Skipped 2 previous similar messages [ 7646.176873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7646.193939] Lustre: Skipped 2 previous similar messages [ 7651.303032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7651.312154] Lustre: Skipped 2 previous similar messages [ 7654.368148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7654.674128] Lustre: server umount lustre-MDT0000 complete [ 7656.426185] LustreError: 180294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7656.463635] LustreError: 180294:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7658.953274] LustreError: 181027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671793 with bad export cookie 3834666047086088234 [ 7658.959554] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7658.967643] LustreError: 181027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7660.575323] LustreError: 181022:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000042998f0b x1875558184025088/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7660.703445] Lustre: server umount lustre-MDT0001 complete [ 7674.558758] Lustre: server umount lustre-OST0000 complete [ 7684.911188] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 7695.858725] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 7729.995680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7730.187263] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7730.211069] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7730.268111] Lustre: lustre-MDT0000: new disk, initializing [ 7730.379566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7730.394673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7734.453121] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7744.775850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7744.857593] Lustre: 185635:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7744.901706] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7744.905081] Lustre: Skipped 1 previous similar message [ 7744.980704] Lustre: lustre-MDT0001: new disk, initializing [ 7745.066036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7745.114041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7745.132996] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7749.707807] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7758.169619] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7763.859909] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7771.228969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7771.560421] Lustre: lustre-OST0000: new disk, initializing [ 7771.571160] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7771.580086] Lustre: 187602:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7771.693740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7778.057965] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7778.896212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7778.905556] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7779.052807] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7787.369125] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7792.076402] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7792.300896] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7797.024991] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7797.374977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7799.778704] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7799.786484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7799.787969] LustreError: Skipped 1 previous similar message [ 7799.826645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7804.899871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7804.909575] Lustre: Skipped 2 previous similar messages [ 7805.586314] Lustre: server umount lustre-OST0000 complete [ 7810.016509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7810.027643] Lustre: Skipped 1 previous similar message [ 7815.140752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7815.148177] Lustre: Skipped 3 previous similar messages [ 7815.625718] Lustre: server umount lustre-MDT0000 complete [ 7819.856570] LustreError: 185626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671954 with bad export cookie 3834666047086088948 [ 7819.863578] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7819.865078] LustreError: 185626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7820.276679] Lustre: server umount lustre-MDT0001 complete [ 7829.746521] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 01:19:24 (1788671964) [ 7847.277296] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 7849.941185] Key type lgssc unregistered [ 7850.216338] LNet: 190548:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7850.227904] LNetError: 190548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7850.245695] LNet: Removed LNI 192.168.201.122@tcp [ 7851.002078] Key type .llcrypt unregistered [ 7851.004654] Key type ._llcrypt unregistered [ 7870.240990] Key type ._llcrypt registered [ 7870.247408] Key type .llcrypt registered [ 7870.394488] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 7872.034551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7872.050147] alg: No test for adler32 (adler32-zlib) [ 7873.194633] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 7873.458423] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 7875.121158] Key type lgssc registered [ 7876.547807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7886.392272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7886.411728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7888.143664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7893.480082] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7897.574377] LustreError: 191764:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7897.607430] LustreError: 191764:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7902.711018] LustreError: 191765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7902.859379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7903.379508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7908.364362] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7916.124869] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7923.418551] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7932.218119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7932.356467] LustreError: 193638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7932.569427] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7932.576816] LustreError: 193631:0:(llog.c:1696:llog_backup()) MGC192.168.201.122@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7932.587881] Lustre: 193631:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.122@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7932.774979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7941.223632] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7951.095891] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7954.404629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7954.409296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7954.424127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7958.505061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7958.505511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7958.527875] Lustre: Skipped 1 previous similar message [ 7960.073159] Lustre: server umount lustre-MDT0000 complete [ 7963.622609] LustreError: 191761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7963.663616] LustreError: 191761:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7964.980050] LustreError: 191745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672099 with bad export cookie 4748601440174689708 [ 7965.001832] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7965.012965] LustreError: 191745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7965.551706] Lustre: server umount lustre-MDT0001 complete [ 7981.310399] Lustre: server umount lustre-OST0000 complete [ 7990.865626] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 8002.016332] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 8038.779230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8039.080679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8039.116893] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8039.175295] Lustre: lustre-MDT0000: new disk, initializing [ 8039.311631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8039.340298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8043.877812] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8054.657730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8054.764075] Lustre: 197160:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8054.835760] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8054.837359] Lustre: Skipped 1 previous similar message [ 8054.842054] LustreError: 197942:0:(llog.c:1696:llog_backup()) MGC192.168.201.122@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8054.847077] LustreError: 197942:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8054.851593] Lustre: 197942:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.122@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8054.863792] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8054.867309] Lustre: Skipped 1 previous similar message [ 8054.951102] Lustre: lustre-MDT0001: new disk, initializing [ 8055.035303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8055.065788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8055.080153] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8059.636761] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8069.102223] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8074.425977] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8081.415278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8081.593789] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8081.597561] Lustre: Skipped 1 previous similar message [ 8081.600667] LustreError: 199105:0:(llog.c:1696:llog_backup()) MGC192.168.201.122@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8081.609720] LustreError: 199105:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8081.614656] Lustre: 199105:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.122@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8081.621549] Lustre: 199105:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8081.643812] Lustre: lustre-OST0000: new disk, initializing [ 8081.648460] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8081.654070] Lustre: 199128:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8081.752597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8087.711611] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8088.631483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8088.650545] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8088.787841] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8097.222482] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8102.403760] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8102.613594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8108.377826] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8108.580230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8114.661643] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8114.674896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8114.700443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8114.725221] Lustre: Skipped 3 previous similar messages [ 8116.850917] Lustre: server umount lustre-OST0000 complete [ 8121.825964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8121.844610] LustreError: Skipped 1 previous similar message [ 8121.854958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8121.879434] Lustre: Skipped 1 previous similar message [ 8121.893309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8127.872410] Lustre: server umount lustre-MDT0000 complete [ 8130.018177] LustreError: 197166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8130.071184] LustreError: 197166:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8132.337357] LustreError: 197150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672267 with bad export cookie 4748601440174690429 [ 8132.341361] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8132.607911] Lustre: server umount lustre-MDT0001 complete [ 8142.170168] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 01:24:36 (1788672276) [ 8160.431641] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 8169.926373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8170.165063] LustreError: 202260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8170.200209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8173.524411] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8181.267230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8185.822397] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8188.535944] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8195.122561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8200.934546] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8209.101702] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8209.306887] Lustre: lustre-OST0001: new disk, initializing [ 8209.311968] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8209.328943] Lustre: 204567:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8209.437775] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8209.443525] Lustre: Skipped 2 previous similar messages [ 8216.525466] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8217.774377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8217.782230] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8217.878207] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8227.977376] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8238.959316] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8241.385567] Lustre: DEBUG MARKER: ADD OST3 [ 8247.413804] loop: module loaded [ 8247.530232] loop0: detected capacity change from 0 to 204800000 [ 8247.732276] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8247.760627] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8247.772735] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8255.497081] Lustre: DEBUG MARKER: STOP OST3 [ 8256.343080] loop0: detected capacity change from 0 to 204800000 [ 8263.783300] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8264.060802] Lustre: lustre-OST0003: new disk, initializing [ 8264.067152] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8264.071892] Lustre: 206642:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8265.876558] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8265.986026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8265.994704] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8266.032187] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8266.587176] Lustre: Failing over lustre-OST0003 [ 8266.997936] Lustre: server umount lustre-OST0003 complete [ 8267.938784] LustreError: 206805:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8270.781482] LustreError: 205095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8270.821682] LustreError: 205095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8271.329129] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8271.340189] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8271.369616] Lustre: Skipped 2 previous similar messages [ 8271.946219] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8282.910731] Lustre: DEBUG MARKER: START OST3 again [ 8289.433150] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8289.893856] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8289.902130] Lustre: Skipped 1 previous similar message [ 8289.915878] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8291.205669] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8291.899791] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8291.906994] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8295.961970] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8318.073922] Lustre: Failing over lustre-OST0003 [ 8318.198539] Lustre: server umount lustre-OST0003 complete [ 8320.503274] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8320.505480] LustreError: 207934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8320.513049] Lustre: Skipped 1 previous similar message [ 8320.539254] LustreError: 207934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 8335.855627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8335.858388] Lustre: Skipped 2 previous similar messages [ 8340.457467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8340.472521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8340.492087] Lustre: Skipped 3 previous similar messages [ 8342.125862] Lustre: server umount lustre-MDT0000 complete [ 8345.420142] LustreError: 202242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672480 with bad export cookie 4748601440174691311 [ 8345.426132] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8345.428632] LustreError: 202242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8346.083138] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8346.112926] Lustre: Skipped 7 previous similar messages [ 8351.981601] Lustre: server umount lustre-MDT0001 complete [ 8356.101867] Lustre: server umount lustre-OST0000 complete [ 8361.337825] Lustre: server umount lustre-OST0001 complete [ 8369.428633] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 8380.361256] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 8417.195709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8417.602701] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8417.667797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8417.796463] Lustre: lustre-MDT0000: new disk, initializing [ 8417.907193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8417.923874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8422.277876] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8432.529965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8432.712427] Lustre: 212177:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8433.007568] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8437.328640] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8449.019707] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8455.744264] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8463.546202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8463.787609] Lustre: lustre-OST0000: new disk, initializing [ 8463.790497] Lustre: Skipped 1 previous similar message [ 8463.792609] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8463.796578] Lustre: Skipped 2 previous similar messages [ 8463.801190] Lustre: 214151:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8465.297260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8465.315477] Lustre: Skipped 1 previous similar message [ 8465.321260] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8465.422622] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8470.953881] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8480.774500] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8485.644090] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8485.862763] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8490.757224] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8491.024207] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8496.101610] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8496.111213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8496.115259] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8496.116675] Lustre: Skipped 3 previous similar messages [ 8496.134602] Lustre: Skipped 3 previous similar messages [ 8507.359265] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8507.560213] Lustre: server umount lustre-OST0000 complete [ 8517.679575] Lustre: server umount lustre-MDT0000 complete [ 8520.930918] LustreError: 212171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672655 with bad export cookie 4748601440174693607 [ 8520.936750] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8520.943992] LustreError: 212171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8521.284873] Lustre: server umount lustre-MDT0001 complete [ 8524.549467] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 01:30:59 (1788672659) [ 8541.356818] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 8553.453407] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 8589.822756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8589.990720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8590.007754] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8590.071804] Lustre: lustre-MDT0000: new disk, initializing [ 8590.133586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8594.848995] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8606.448796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8606.598666] Lustre: 218762:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8606.874214] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8613.054572] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8623.495290] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8629.973363] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8637.460735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8637.801433] Lustre: 220731:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8639.726766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8639.830511] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8644.171519] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8653.309882] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8658.694320] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8658.845204] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8663.885419] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8664.115638] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8670.688159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8670.692253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8670.707929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8670.726916] Lustre: Skipped 3 previous similar messages [ 8670.745458] Lustre: Skipped 9 previous similar messages [ 8672.439139] Lustre: server umount lustre-OST0000 complete [ 8682.549597] Lustre: server umount lustre-MDT0000 complete [ 8685.858395] LustreError: 218753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672820 with bad export cookie 4748601440174694489 [ 8685.868958] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8685.878040] LustreError: 218753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8686.220410] Lustre: server umount lustre-MDT0001 complete [ 8695.316629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8695.787067] LustreError: 222947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8695.808193] LustreError: 222947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 8695.830478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8695.837739] Lustre: Skipped 5 previous similar messages [ 8701.005442] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8710.677658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8717.029706] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8725.847781] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8732.812284] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8741.155788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8749.122322] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8757.498967] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8775.648504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8775.653405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8775.655403] LustreError: Skipped 1 previous similar message [ 8775.680436] Lustre: Skipped 4 previous similar messages [ 8779.505706] Lustre: server umount lustre-OST0000 complete [ 8789.590500] Lustre: server umount lustre-MDT0000 complete [ 8793.987760] LustreError: 222927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672928 with bad export cookie 4748601440174695371 [ 8794.008286] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8794.422248] Lustre: server umount lustre-MDT0001 complete [ 8898.037930] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 8900.623434] Key type lgssc unregistered [ 8900.916791] LNet: 226630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8900.925229] LNetError: 226630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8900.964175] LNet: Removed LNI 192.168.201.122@tcp [ 8901.637203] Key type .llcrypt unregistered [ 8901.638703] Key type ._llcrypt unregistered [ 8928.786561] Key type ._llcrypt registered [ 8928.788740] Key type .llcrypt registered [ 8928.878178] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 8945.950637] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 8946.862513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8947.048024] alg: No test for adler32 (adler32-zlib) [ 8948.255140] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 8948.492657] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 8950.151185] Key type lgssc registered [ 8951.134637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8990.471190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8990.527410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8991.811089] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8991.859687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8991.940454] Lustre: lustre-MDT0000: new disk, initializing [ 8992.026559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8992.046132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8997.506485] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 9009.156218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9009.281807] Lustre: 229982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9009.327164] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9009.330615] Lustre: Skipped 1 previous similar message [ 9009.413525] Lustre: lustre-MDT0001: new disk, initializing [ 9009.532206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9009.579806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9009.603831] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9014.501189] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 9024.893309] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9030.877506] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9038.153803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9038.515474] Lustre: lustre-OST0000: new disk, initializing [ 9038.519468] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9038.531370] Lustre: 231949:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9038.649290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9045.746819] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 9048.139039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9048.154048] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9048.303198] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9056.332267] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9061.623130] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9061.813130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9066.881497] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9067.039564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9069.025244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9069.032017] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9069.039447] LustreError: Skipped 1 previous similar message [ 9069.044198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9069.053296] Lustre: Skipped 1 previous similar message [ 9069.073625] Lustre: Skipped 1 previous similar message [ 9074.158311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9074.172174] Lustre: Skipped 1 previous similar message [ 9079.269620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9079.279069] Lustre: Skipped 1 previous similar message [ 9083.361269] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9083.491486] Lustre: server umount lustre-OST0000 complete [ 9089.507597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9089.515794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9089.534066] Lustre: Skipped 1 previous similar message [ 9089.549456] Lustre: Skipped 1 previous similar message [ 9094.627695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9102.303874] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9102.610283] Lustre: server umount lustre-MDT0000 complete [ 9106.878661] LustreError: 229972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673241 with bad export cookie 12879084227537135129 [ 9106.886182] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9106.895094] LustreError: 229972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9107.310356] Lustre: server umount lustre-MDT0001 complete [ 9126.778475] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 9137.041408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9137.382727] LustreError: 234886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9137.399138] LustreError: 234886:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9137.439828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9141.475229] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 9142.768673] LustreError: 234887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9147.875921] LustreError: 234886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9150.223865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9150.540151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9154.913653] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 9158.214617] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9165.406335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9165.853913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9172.638370] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 9181.057295] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9181.214828] Lustre: lustre-OST0001: new disk, initializing [ 9181.218600] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9181.224226] Lustre: 237193:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9187.462736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9187.473401] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9187.577262] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9188.856149] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 9200.843912] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9209.577383] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9225.862759] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 01:42:40 (1788673360) [ 9228.785533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9228.799346] Lustre: Skipped 1 previous similar message [ 9228.801591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9228.820421] Lustre: Skipped 6 previous similar messages [ 9234.253202] Lustre: server umount lustre-MDT0000 complete [ 9238.191262] LustreError: 237961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673373 with bad export cookie 12879084227537136004 [ 9238.193585] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9238.202924] LustreError: 237961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9238.684044] Lustre: server umount lustre-MDT0001 complete [ 9253.692587] Lustre: server umount lustre-OST0000 complete [ 9254.559501] Lustre: 228202:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788673373/real 1788673373] req@ffff9f867cf7bb80 x1875559666793088/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788673389 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9254.612572] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9254.624632] Lustre: Skipped 1 previous similar message [ 9256.914859] Lustre: server umount lustre-OST0001 complete [ 9263.942409] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 9276.863247] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 9317.720360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9317.934581] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9317.963969] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9318.051964] Lustre: lustre-MDT0000: new disk, initializing [ 9318.187550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9318.205081] Lustre: Skipped 1 previous similar message [ 9318.224395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9323.088443] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 9336.762262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9336.947802] Lustre: 241140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9337.008219] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9337.019457] Lustre: Skipped 1 previous similar message [ 9337.124896] Lustre: lustre-MDT0001: new disk, initializing [ 9337.205914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9337.215678] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9342.099185] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 9355.312946] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9363.320834] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9371.258067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9371.561882] Lustre: lustre-OST0000: new disk, initializing [ 9371.565752] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9371.576772] Lustre: 243111:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9371.712510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9371.717725] Lustre: Skipped 1 previous similar message [ 9373.473284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9373.484780] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9373.652394] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9378.970981] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 9389.211483] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9394.217979] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9394.546365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9399.299693] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9399.568358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9404.386562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9404.391727] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9404.394898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9404.402541] Lustre: Skipped 1 previous similar message [ 9404.418476] Lustre: Skipped 5 previous similar messages [ 9407.845765] Lustre: server umount lustre-OST0000 complete [ 9414.112373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9414.129133] LustreError: Skipped 1 previous similar message [ 9414.132757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9426.399139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9426.530947] Lustre: server umount lustre-MDT0000 complete [ 9429.990555] LustreError: 241151:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9430.007640] LustreError: 241151:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9430.328401] LustreError: 241132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673565 with bad export cookie 12879084227537138146 [ 9430.334708] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9430.338892] LustreError: 241132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9430.761951] Lustre: server umount lustre-MDT0001 complete [ 9441.415164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9442.299381] LustreError: 245331:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9442.467671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9448.320367] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 9453.030997] LustreError: 245332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9453.053822] LustreError: 245332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9460.660986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9467.512107] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 9475.607190] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9483.109866] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9489.896982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9496.532762] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 9504.448803] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9654.248656] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9654.254587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9654.272788] Lustre: Skipped 1 previous similar message [ 9654.280959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9654.283592] Lustre: Skipped 6 previous similar messages [ 9657.660143] Lustre: server umount lustre-OST0000 complete [ 9666.017064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9666.025108] LustreError: Skipped 1 previous similar message [ 9676.255870] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9676.463996] Lustre: server umount lustre-MDT0000 complete [ 9676.771669] LustreError: 246429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9676.787924] LustreError: 246429:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9680.756415] LustreError: 245313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673815 with bad export cookie 12879084227537139028 [ 9680.774953] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9681.117474] Lustre: server umount lustre-MDT0001 complete [ 9771.786578] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 9774.912724] Key type lgssc unregistered [ 9775.273681] LNet: 248939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9775.280334] LNetError: 248939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9775.296951] LNet: Removed LNI 192.168.201.122@tcp [ 9776.095572] Key type .llcrypt unregistered [ 9776.100807] Key type ._llcrypt unregistered [ 9806.875222] Key type ._llcrypt registered [ 9806.877571] Key type .llcrypt registered [ 9807.040941] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 9825.193916] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 9826.417031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9826.615224] alg: No test for adler32 (adler32-zlib) [ 9827.709565] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 9828.003918] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 9829.727166] Key type lgssc registered [ 9831.089813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9881.879464] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 9892.517821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9892.551574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9893.858398] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9893.891275] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9894.027783] Lustre: lustre-MDT0000: new disk, initializing [ 9894.140723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9894.185258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9899.343964] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 9910.611979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9910.764913] Lustre: 253006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9910.839383] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9910.844031] Lustre: Skipped 1 previous similar message [ 9910.965798] Lustre: lustre-MDT0001: new disk, initializing [ 9911.079464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9911.170979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9911.179537] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9916.623364] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 9923.005440] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9931.111835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9931.506164] Lustre: lustre-OST0000: new disk, initializing [ 9931.512187] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9931.522185] Lustre: 254640:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9931.704316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9939.043922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9939.059913] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9939.270742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9940.352348] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 9953.542208] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9953.793374] Lustre: lustre-OST0001: new disk, initializing [ 9953.798240] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9953.806494] Lustre: 255512:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9953.958779] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9960.661135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9960.691712] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9960.896215] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9961.905381] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 9975.197921] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9983.240859] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9996.769487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9996.806569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9996.812812] Lustre: Skipped 1 previous similar message [ 9996.851773] Lustre: Skipped 3 previous similar messages [10001.752374] Lustre: server umount lustre-MDT0000 complete [10007.023739] LustreError: 255520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10007.053021] LustreError: 255520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [10007.378917] LustreError: 254638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674142 with bad export cookie 9742315072566971888 [10007.383303] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10007.403850] LustreError: 254638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10008.534486] Lustre: server umount lustre-MDT0001 complete [10025.325994] Lustre: server umount lustre-OST0000 complete [10028.518176] Lustre: 250514:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788674147/real 1788674147] req@ffff9f8548a81f80 x1875560588461184/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788674163 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10028.545036] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10028.558689] Lustre: Skipped 2 previous similar messages [10030.113419] Lustre: server umount lustre-OST0001 complete [10044.376824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10053.221732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10061.624949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10071.773402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10094.929847] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [10106.306725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10106.335343] Lustre: lustre-MDT0000: reset Object Index mappings [10106.849125] LustreError: 260523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10106.954274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10111.856722] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [10111.975206] LustreError: 260524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10117.098099] LustreError: 260523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10120.015996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10120.038508] Lustre: lustre-MDT0001: reset Object Index mappings [10120.451539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10124.631148] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [10127.430361] Lustre: 261668:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10134.346309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10141.066856] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [10146.024384] LustreError: 262021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10149.412477] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10149.636267] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10149.643532] Lustre: Skipped 1 previous similar message [10151.681890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10151.684246] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10155.032873] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [10163.423359] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10167.612042] Lustre: 263537:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10172.384817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10172.394619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10172.398290] Lustre: Skipped 2 previous similar messages [10174.572348] Lustre: server umount lustre-MDT0000 complete [10177.509605] LustreError: 260524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10177.532947] LustreError: 260524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10178.755577] LustreError: 260503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674313 with bad export cookie 9742315072566978678 [10178.761563] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10178.762116] LustreError: 260503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10179.412564] Lustre: server umount lustre-MDT0001 complete [10194.803641] Lustre: server umount lustre-OST0000 complete [10199.007160] Lustre: 250513:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788674317/real 1788674317] req@ffff9f854264c700 x1875560588508672/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788674333 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10199.032869] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10199.049584] Lustre: Skipped 3 previous similar messages [10200.037256] Lustre: 250511:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788674318/real 1788674318] req@ffff9f854264c000 x1875560588508928/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788674334 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10200.714190] Lustre: server umount lustre-OST0001 complete [10219.667595] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 01:59:14 (1788674354) [10239.544929] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [10255.094966] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [10292.521712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10292.740796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10292.762604] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10292.832544] Lustre: lustre-MDT0000: new disk, initializing [10292.902970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10292.913825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10297.900505] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [10308.709157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10308.791612] Lustre: 267814:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10308.843977] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10308.847785] Lustre: Skipped 1 previous similar message [10308.925522] Lustre: lustre-MDT0001: new disk, initializing [10309.058366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10309.072576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10312.874973] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [10321.094489] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10326.080168] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10332.801986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10333.192466] Lustre: lustre-OST0000: new disk, initializing [10333.201740] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10333.210342] Lustre: 269793:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10334.986855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10335.010315] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10335.253089] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10339.410576] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [10351.078553] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10357.673611] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10358.089656] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10363.818518] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10364.078673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10371.046670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10371.068578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10371.070323] Lustre: Skipped 1 previous similar message [10371.093273] Lustre: Skipped 1 previous similar message [10376.166403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10376.175393] Lustre: Skipped 1 previous similar message [10380.255141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10380.562576] Lustre: server umount lustre-OST0000 complete [10385.889649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10385.899588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10385.918163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10396.640910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10396.648795] Lustre: Skipped 5 previous similar messages [10399.199311] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10399.499595] Lustre: server umount lustre-MDT0000 complete [10401.762762] LustreError: 268614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10401.790486] LustreError: 268614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10403.200348] LustreError: 267807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674538 with bad export cookie 9742315072566980771 [10403.202335] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10403.220023] LustreError: 267807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10403.496500] Lustre: server umount lustre-MDT0001 complete [10425.193322] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [10435.793382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10436.158350] LustreError: 272731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10436.208901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10436.217948] Lustre: Skipped 2 previous similar messages [10441.032248] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [10451.585549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10456.644960] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [10459.385448] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10466.232358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10473.286475] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [10482.733986] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10482.898543] Lustre: lustre-OST0001: new disk, initializing [10482.901795] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10482.906745] Lustre: 275040:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10484.791098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10484.794441] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10485.022818] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10490.893388] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [10501.686863] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10505.566934] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10510.670449] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10510.707440] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10523.618305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10523.623530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10523.644075] Lustre: Skipped 1 previous similar message [10523.651373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10527.927502] Lustre: server umount lustre-MDT0000 complete [10531.260971] LustreError: 272711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674666 with bad export cookie 9742315072566981646 [10531.269851] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10531.272656] LustreError: 272711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10531.303302] LustreError: 272726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10531.323400] LustreError: 272726:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10531.743517] Lustre: server umount lustre-MDT0001 complete [10537.964831] Lustre: server umount lustre-OST0000 complete [10542.512903] Lustre: server umount lustre-OST0001 complete [10567.910119] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [10579.303340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10579.800679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10579.808426] Lustre: Skipped 3 previous similar messages [10584.173847] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [10592.214916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10597.411770] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [10600.147920] Lustre: 279549:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10607.010699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10613.808804] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [10623.772110] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10625.091702] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10625.121595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10631.780881] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [10640.509947] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10644.838623] Lustre: 281423:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10650.945329] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10651.047436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10665.583706] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 02:06:39 (1788674799) [10681.317062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10681.320178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10681.320188] Lustre: Skipped 6 previous similar messages [10681.347297] Lustre: Skipped 7 previous similar messages [10687.194378] Lustre: server umount lustre-MDT0000 complete [10690.959603] LustreError: 278390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674825 with bad export cookie 9742315072566988541 [10690.972800] LustreError: 278390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10690.979926] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10691.554688] LustreError: 281437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10691.580387] LustreError: 281437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10705.381910] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10705.813445] Lustre: server umount lustre-MDT0001 complete [10716.808525] Lustre: server umount lustre-OST0000 complete [10721.684410] Lustre: server umount lustre-OST0001 complete [10730.872215] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [10742.160339] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [10797.119941] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [10809.206866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10809.539124] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10809.558484] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10809.640857] Lustre: lustre-MDT0000: new disk, initializing [10809.767396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10815.099927] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [10826.152870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10826.355437] Lustre: 285957:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10826.820345] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10832.263327] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [10845.635442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10845.916962] Lustre: 287590:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10846.065724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10846.070207] Lustre: Skipped 5 previous similar messages [10847.740779] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10854.455542] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [10866.831863] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10867.168764] Lustre: 288460:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10868.441350] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10868.457117] Lustre: Skipped 1 previous similar message [10868.615661] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10874.170523] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [10885.644527] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10889.337779] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10889.342376] Lustre: Skipped 1 previous similar message [10905.611646] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 02:10:40 (1788675040) [10908.645341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10908.658319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10908.669087] Lustre: Skipped 3 previous similar messages [10908.690196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10908.702897] Lustre: Skipped 13 previous similar messages [10909.939776] Lustre: server umount lustre-MDT0000 complete [10913.928509] LustreError: 285950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675048 with bad export cookie 9742315072566996031 [10913.932339] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10913.946432] LustreError: 285950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10914.449812] Lustre: server umount lustre-MDT0001 complete [10929.319191] Lustre: server umount lustre-OST0000 complete [10930.911191] Lustre: 250511:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788675049/real 1788675049] req@ffff9f866f419c00 x1875560588779008/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788675065 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10933.038428] Lustre: server umount lustre-OST0001 complete [10941.843786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10947.555899] LustreError: 290707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10947.583898] LustreError: 290707:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10947.947137] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [10957.532603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10962.642129] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [10970.368537] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10976.700296] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10984.076636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10990.892927] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [10999.334371] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11019.679737] loop1: detected capacity change from 0 to 204800000 [11020.102123] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11020.123891] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11020.159150] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11026.932250] loop1: detected capacity change from 0 to 204800000 [11033.339215] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11033.499667] LustreError: 290697:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs16366: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11033.531563] LustreError: 290697:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs16366 [11033.533763] LustreError: 290697:0:(mgs_handler.c:517:mgs_target_reg()) fs16366-MDT0001: Error checking target index: rc = -13 [11033.549159] LustreError: fs16366-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11033.598457] LustreError: 294424:0:(tgt_mount.c:2619:server_fill_super()) Unable to start targets: -13 [11033.610514] LustreError: 294424:0:(tgt_mount.c:2134:server_put_super()) no obd fs16366-MDT0001 [11033.618256] LustreError: 294424:0:(tgt_mount.c:116:server_deregister_mount()) fs16366-MDT0001 not registered [11033.686799] Lustre: server umount fs16366-MDT0001 complete [11033.701976] LustreError: 294424:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11045.614781] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11045.906648] Lustre: Setting parameter fs16366-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs16366-MDT0001 [11046.031888] Lustre: srv-fs16366-MDT0001: No data found on store. Initialize space. [11046.045383] Lustre: Skipped 4 previous similar messages [11046.261242] Lustre: fs16366-MDT0001: new disk, initializing [11046.271473] Lustre: Skipped 3 previous similar messages [11046.497452] Lustre: srv-fs16366-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11046.513300] Lustre: 294892:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16366-MDT0001-mdtlov: can't create new llog: rc = -115 [11046.536048] LustreError: 294892:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16366-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11050.379573] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [11056.717082] Lustre: server umount fs16366-MDT0001 complete [11069.414777] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11071.682324] loop1: detected capacity change from 0 to 204800000 [11077.901553] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11078.376416] Lustre: 296460:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16366-MDT0001-mdtlov: can't create new llog: rc = -115 [11078.386254] LustreError: 296460:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16366-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11082.835308] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [11084.776966] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11086.946231] Lustre: server umount fs16366-MDT0001 complete [11096.544509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11096.550254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11096.557831] Lustre: Skipped 4 previous similar messages [11096.562071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11096.564730] Lustre: Skipped 4 previous similar messages [11110.369260] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11110.592700] Lustre: server umount lustre-MDT0000 complete [11118.699891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11118.780268] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11123.308889] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [11124.220362] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11131.636503] loop1: detected capacity change from 0 to 204800000 [11137.710536] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11138.206820] Lustre: 298908:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16366-MDT0001-mdtlov: can't create new llog: rc = -115 [11138.213984] LustreError: 298908:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16366-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11144.524219] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [11150.533914] Lustre: server umount fs16366-MDT0001 complete [11175.397146] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11179.776334] Lustre: server umount lustre-OST0000 complete [11190.316320] Lustre: server umount lustre-MDT0000 complete [11194.493920] LustreError: 291448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675329 with bad export cookie 9742315072566998712 [11194.502663] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11194.520620] LustreError: 291448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11195.073414] Lustre: server umount lustre-MDT0001 complete [11207.576875] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [11210.750703] Key type lgssc unregistered [11211.011842] LNet: 301253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11211.025318] LNetError: 301253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11211.052331] LNet: Removed LNI 192.168.201.122@tcp [11212.145269] Key type .llcrypt unregistered [11212.148137] Key type ._llcrypt unregistered [11238.834857] Key type ._llcrypt registered [11238.838524] Key type .llcrypt registered [11238.948873] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [11240.126856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11240.149087] alg: No test for adler32 (adler32-zlib) [11241.191596] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [11241.460489] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [11243.119241] Key type lgssc registered [11244.286993] Lustre: Echo OBD driver; http://www.lustre.org/ [11252.213944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11252.249155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11253.622134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11258.276073] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [11263.973085] LustreError: 302521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11263.990472] LustreError: 302521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11266.620160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11266.926819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11270.722536] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [11278.210966] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11285.074643] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11293.385881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11293.786267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11298.928179] LustreError: 304350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11300.715610] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [11304.421538] LustreError: 304352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11309.560080] LustreError: 304350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11309.869342] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11316.722997] LustreError: 304350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11316.731463] LustreError: 304350:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11326.948793] LustreError: 304618:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11326.977765] LustreError: 304618:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11331.509309] loop1: detected capacity change from 0 to 204800000 [11331.746346] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11331.772207] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11331.814996] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11337.233833] loop1: detected capacity change from 0 to 204800000 [11342.640468] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11342.874534] Lustre: fs27436-OST0000: new disk, initializing [11342.878733] Lustre: srv-fs27436-OST0000: No data found on store. Initialize space. [11342.886365] Lustre: 306257:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11342.999960] Lustre: fs27436-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11347.430442] LustreError: 304618:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11347.448147] LustreError: 304618:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [11349.442172] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [11357.478398] Lustre: server umount fs27436-OST0000 complete [11369.212629] loop1: detected capacity change from 0 to 204800000 [11374.261486] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11374.446381] Lustre: fs27436-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11378.153145] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11379.832381] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [11383.275229] LustreError: 307325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11383.300967] LustreError: 307325:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [11387.730578] loop2: detected capacity change from 0 to 204800000 [11388.054776] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11388.071410] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11388.080892] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11392.740696] loop2: detected capacity change from 0 to 204800000 [11393.512284] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11398.330499] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11398.458235] LustreError: fs27436-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11398.470162] LustreError: fs27436-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11398.485528] LustreError: 309299:0:(tgt_mount.c:2619:server_fill_super()) Unable to start targets: -13 [11398.496840] LustreError: 309299:0:(tgt_mount.c:2134:server_put_super()) no obd fs27436-OST0001 [11398.515258] LustreError: 309299:0:(tgt_mount.c:116:server_deregister_mount()) fs27436-OST0001 not registered [11398.566945] Lustre: server umount fs27436-OST0001 complete [11398.570391] LustreError: 309299:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11408.557231] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11408.777650] Lustre: fs27436-OST0001: new disk, initializing [11408.783630] Lustre: srv-fs27436-OST0001: No data found on store. Initialize space. [11408.793612] Lustre: 309852:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11408.909497] Lustre: fs27436-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11414.724643] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [11422.061681] Lustre: server umount fs27436-OST0000 complete [11429.979430] Lustre: server umount fs27436-OST0001 complete [11441.867492] loop1: detected capacity change from 0 to 204800000 [11448.023913] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11448.274490] Lustre: fs27436-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11449.826529] LustreError: 304353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11449.844954] LustreError: 304353:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11455.934794] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [11464.060393] loop2: detected capacity change from 0 to 204800000 [11470.186759] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11477.140903] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [11486.490358] Lustre: server umount fs27436-OST0000 complete [11495.279626] Lustre: server umount fs27436-OST0001 complete [11509.218770] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11509.229481] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11509.242382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11511.271856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11511.273469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11511.279755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11512.571848] Lustre: server umount lustre-OST0000 complete [11516.389519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11516.399207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11516.410034] Lustre: Skipped 1 previous similar message [11516.417770] Lustre: Skipped 1 previous similar message [11521.505646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11521.522882] Lustre: Skipped 1 previous similar message [11522.420629] Lustre: server umount lustre-MDT0000 complete [11525.606791] LustreError: 302504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675660 with bad export cookie 17167362626100716727 [11525.615573] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11525.616578] LustreError: 302504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11525.934564] Lustre: server umount lustre-MDT0001 complete [11535.817115] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [11538.057351] Key type lgssc unregistered [11538.352989] LNet: 315770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11538.363869] LNetError: 315770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11538.384993] LNet: Removed LNI 192.168.201.122@tcp [11539.238888] Key type .llcrypt unregistered [11539.240741] Key type ._llcrypt unregistered [11571.187897] Key type ._llcrypt registered [11571.190430] Key type .llcrypt registered [11571.309926] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [11586.903856] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [11588.083439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11588.123767] alg: No test for adler32 (adler32-zlib) [11589.168602] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [11589.509862] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [11591.183845] Key type lgssc registered [11592.561654] Lustre: Echo OBD driver; http://www.lustre.org/ [11630.506963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11630.536800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11631.731919] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11631.785066] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11631.837506] Lustre: lustre-MDT0000: new disk, initializing [11631.910664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11631.953508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11636.220517] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [11647.190308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11647.358166] Lustre: 319628:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11647.429808] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11647.432833] Lustre: Skipped 1 previous similar message [11647.498455] Lustre: lustre-MDT0001: new disk, initializing [11647.592199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11647.631308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11647.646975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11652.488593] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [11661.652905] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11667.433950] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11674.871461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11675.050326] Lustre: lustre-OST0000: new disk, initializing [11675.052863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11675.058052] Lustre: 321598:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11675.137270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11681.378636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11681.390777] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11681.545072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11682.233759] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [11693.081597] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11695.505537] LustreError: 317801:0:(lmv_obd.c:211:lmv_notify()) activation of lustre-MDT0001_UUID failed: -22 [11695.540690] Lustre: Mounted lustre-client - version 2.17.58_4_g3b30273 [11696.558528] Lustre: Unmounted lustre-client [11710.945728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11710.949575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11710.991353] Lustre: Skipped 1 previous similar message [11711.014106] Lustre: Skipped 1 previous similar message [11715.683191] Lustre: server umount lustre-OST0000 complete [11721.193893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11721.198761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11721.221593] Lustre: Skipped 1 previous similar message [11721.235374] Lustre: Skipped 1 previous similar message [11726.311411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11726.320208] Lustre: Skipped 1 previous similar message [11727.188501] Lustre: server umount lustre-MDT0000 complete [11730.491388] LustreError: 319620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675865 with bad export cookie 8429365265350956773 [11730.497668] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11730.500147] LustreError: 319620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11730.754950] Lustre: server umount lustre-MDT0001 complete [11742.662560] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [11745.987940] Key type lgssc unregistered [11746.208526] LNet: 1 local NIs in recovery (showing 1): 192.168.201.122@tcp [11746.312387] LNet: 323758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11746.318858] LNetError: 323758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11826.216286] LNet: Removed LNI 192.168.201.122@tcp [11827.289655] Key type .llcrypt unregistered [11827.291473] Key type ._llcrypt unregistered [11848.694745] Key type ._llcrypt registered [11848.697825] Key type .llcrypt registered [11848.881682] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [11869.498332] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [11870.651626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11870.670657] alg: No test for adler32 (adler32-zlib) [11871.741930] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [11871.937864] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [11873.663417] Key type lgssc registered [11875.247118] Lustre: Echo OBD driver; http://www.lustre.org/ [11910.421366] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 02:27:24 (1788676044) [11917.599970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11917.672809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11918.944539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11919.012873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11919.141070] Lustre: lustre-MDT0000: new disk, initializing [11919.288587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11919.319328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11924.574593] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [11937.670909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11937.821569] Lustre: 326976:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11937.967446] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11937.980252] Lustre: Skipped 1 previous similar message [11938.202744] Lustre: lustre-MDT0001: new disk, initializing [11938.433030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11938.526278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11938.568741] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11945.051294] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [11957.195090] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11963.907914] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11970.363356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11970.588190] Lustre: lustre-OST0000: new disk, initializing [11970.591930] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11970.597314] Lustre: 328942:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11970.682488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11976.766782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11976.780976] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11976.905545] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11977.672051] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [11989.338226] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12013.031508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12013.038767] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12013.055448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12014.388983] Lustre: server umount lustre-OST0000 complete [12020.193884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12020.208399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12020.222279] Lustre: Skipped 1 previous similar message [12020.233140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12020.241320] Lustre: Skipped 1 previous similar message [12023.266052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12023.283771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12025.856026] Lustre: server umount lustre-MDT0000 complete [12030.457589] LustreError: 326968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676165 with bad export cookie 4044049681745954614 [12030.474961] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12030.480558] LustreError: 326968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12031.048574] Lustre: server umount lustre-MDT0001 complete [12047.296335] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [12050.413624] Key type lgssc unregistered [12050.718549] LNet: 330852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12050.737035] LNetError: 330852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12051.751060] LNet: Removed LNI 192.168.201.122@tcp [12052.804373] Key type .llcrypt unregistered [12052.806059] Key type ._llcrypt unregistered [12078.183439] Key type ._llcrypt registered [12078.185795] Key type .llcrypt registered [12078.311220] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [12079.918221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12079.968747] alg: No test for adler32 (adler32-zlib) [12081.033842] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [12081.315453] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [12083.031285] Key type lgssc registered [12084.163267] Lustre: Echo OBD driver; http://www.lustre.org/ [12094.314428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12094.328737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12095.931761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12102.017391] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [12105.189459] LustreError: 332123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12105.218421] LustreError: 332123:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12110.309400] LustreError: 332124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12113.449466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12113.974182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12118.776406] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [12128.189207] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12137.998793] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12147.785986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12148.210434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12154.705985] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [12161.969102] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12176.126424] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12177.710692] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12183.466429] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.22@tcp (stopping) [12185.569939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12185.584640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12185.601329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12188.550577] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.22@tcp (stopping) [12188.562527] Lustre: Skipped 3 previous similar messages [12191.201888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12191.216951] Lustre: Skipped 2 previous similar messages [12196.322766] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12196.323129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12196.370097] Lustre: Skipped 3 previous similar messages [12196.802562] Lustre: server umount lustre-MDT0000 complete [12198.805323] LustreError: 332118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12201.446240] LustreError: 332119:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12201.485189] LustreError: 332119:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12202.107426] LustreError: 332103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676337 with bad export cookie 13864441331454633350 [12202.110576] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12202.127954] LustreError: 332103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12206.562051] LustreError: 333122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12206.563898] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12206.585419] LustreError: 333122:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12206.600412] Lustre: Skipped 2 previous similar messages [12206.602397] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12206.627779] Lustre: Skipped 1 previous similar message [12208.789347] Lustre: server umount lustre-MDT0001 complete [12213.585676] Lustre: server umount lustre-OST0000 complete [12222.189762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12227.122072] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [12230.165214] Lustre: 336308:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.122@tcp [12232.212885] Lustre: server umount MGS complete [12240.482826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12240.790297] LustreError: 336879:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12240.807598] LustreError: 336879:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12240.848118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12245.769420] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [12254.698941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12255.337056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12260.300485] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [12269.137349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12269.557166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12276.509402] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [12284.358507] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12286.035633] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12292.414716] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12294.263160] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12301.303315] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12301.312687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12301.330064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12301.353524] Lustre: Skipped 1 previous similar message [12301.493549] Lustre: server umount lustre-OST0000 complete [12306.410522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12306.414947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12315.779343] Lustre: server umount lustre-MDT0000 complete [12317.669461] LustreError: 337861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12317.696160] LustreError: 337861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [12319.749796] LustreError: 336861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676454 with bad export cookie 13864441331454634183 [12319.774611] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12320.102767] Lustre: server umount lustre-MDT0001 complete [12334.522974] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [12338.606428] Key type lgssc unregistered [12339.228896] LNet: 340314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12339.246649] LNetError: 340314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12339.296911] LNet: Removed LNI 192.168.201.122@tcp [12340.748747] Key type .llcrypt unregistered [12340.762278] Key type ._llcrypt unregistered [12365.940614] Key type ._llcrypt registered [12365.942252] Key type .llcrypt registered [12366.005191] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [12387.690734] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [12389.217317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12389.448524] alg: No test for adler32 (adler32-zlib) [12390.715942] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [12391.069393] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [12392.751200] Key type lgssc registered [12394.167237] Lustre: Echo OBD driver; http://www.lustre.org/ [12433.199402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12433.292031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12434.696664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12434.739814] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12434.898569] Lustre: lustre-MDT0000: new disk, initializing [12435.049944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12435.075358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12440.280845] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [12452.314362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12452.438980] Lustre: 343472:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12452.487993] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12452.492079] Lustre: Skipped 1 previous similar message [12452.561960] Lustre: lustre-MDT0001: new disk, initializing [12452.658483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12452.710295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12452.728724] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12458.741803] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [12470.552665] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12477.927661] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12487.054100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12487.403353] Lustre: lustre-OST0000: new disk, initializing [12487.410783] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12487.431943] Lustre: 345443:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12487.548711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12494.672678] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [12495.453533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12495.470463] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12495.602040] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12505.976300] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12510.642504] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12510.820788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12516.537967] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12516.819541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12521.450046] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12521.462665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12521.474135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12521.493427] Lustre: Skipped 1 previous similar message [12525.391229] Lustre: server umount lustre-OST0000 complete [12529.632681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12529.650449] LustreError: Skipped 1 previous similar message [12529.665308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12529.693581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12529.696716] Lustre: Skipped 1 previous similar message [12531.695462] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12531.721563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12531.757469] Lustre: Skipped 1 previous similar message [12535.753075] Lustre: server umount lustre-MDT0000 complete [12539.697444] LustreError: 343465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676674 with bad export cookie 12472841572031162921 [12539.701590] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12539.711919] LustreError: 343465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12540.161167] Lustre: server umount lustre-MDT0001 complete [12550.995753] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 02:38:05 (1788676685) [12572.498404] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [12585.562622] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [12628.051304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12628.267057] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12628.291315] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12628.351743] Lustre: lustre-MDT0000: new disk, initializing [12628.426543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12628.436914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12633.776562] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [12649.613842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12649.783287] Lustre: 350217:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12649.872180] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12649.880493] Lustre: Skipped 1 previous similar message [12650.004262] Lustre: lustre-MDT0001: new disk, initializing [12650.212222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12650.269134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12650.289477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12656.249544] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [12668.501966] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12676.602630] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12684.945439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12685.169496] Lustre: lustre-OST0000: new disk, initializing [12685.172668] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12685.176356] Lustre: 352187:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12685.261936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12692.533419] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [12692.623538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12692.632871] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12692.779612] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12703.376331] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12744.169994] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12744.172174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12744.181609] LustreError: Skipped 1 previous similar message [12744.205064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12747.450870] Lustre: server umount lustre-OST0000 complete [12752.888584] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12752.904524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12752.921156] Lustre: Skipped 1 previous similar message [12752.929307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12752.942863] Lustre: Skipped 1 previous similar message [12758.156825] Lustre: server umount lustre-MDT0000 complete [12759.528558] LustreError: 350229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12759.548792] LustreError: 350229:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12762.120572] LustreError: 350210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676897 with bad export cookie 12472841572031163796 [12762.122217] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12762.132938] LustreError: 350210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12762.574628] Lustre: server umount lustre-MDT0001 complete [12777.746499] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [12781.370894] Key type lgssc unregistered [12781.741328] LNet: 354358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12781.753173] LNetError: 354358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12781.770630] LNet: Removed LNI 192.168.201.122@tcp [12782.898464] Key type .llcrypt unregistered [12782.900349] Key type ._llcrypt unregistered [12898.591699] Key type ._llcrypt registered [12898.593738] Key type .llcrypt registered [12898.682670] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [12898.969643] Key type .llcrypt unregistered [12898.973056] Key type ._llcrypt unregistered [12993.462385] Key type ._llcrypt registered [12993.466118] Key type .llcrypt registered [12993.612472] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [12993.928482] Key type .llcrypt unregistered [12993.931486] Key type ._llcrypt unregistered [13035.963793] Key type ._llcrypt registered [13035.969875] Key type .llcrypt registered [13036.132408] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [13036.479468] Key type .llcrypt unregistered [13036.488861] Key type ._llcrypt unregistered [13073.437930] Key type ._llcrypt registered [13073.441214] Key type .llcrypt registered [13073.569854] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [13073.963467] Key type .llcrypt unregistered [13073.970227] Key type ._llcrypt unregistered [13162.807132] Key type ._llcrypt registered [13162.818370] Key type .llcrypt registered [13162.957473] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [13163.265270] Key type .llcrypt unregistered [13163.266445] Key type ._llcrypt unregistered [13195.866761] Key type ._llcrypt registered [13195.869547] Key type .llcrypt registered [13195.998670] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [13196.247349] Key type .llcrypt unregistered [13196.253495] Key type ._llcrypt unregistered [13229.567613] Key type ._llcrypt registered [13229.569850] Key type .llcrypt registered [13229.681163] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [13230.023370] Key type .llcrypt unregistered [13230.031499] Key type ._llcrypt unregistered [13263.986648] Key type ._llcrypt registered [13263.989786] Key type .llcrypt registered [13264.116479] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [13264.523717] Key type .llcrypt unregistered [13264.531843] Key type ._llcrypt unregistered [13301.182894] Key type ._llcrypt registered [13301.189376] Key type .llcrypt registered [13301.321175] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [13301.829382] Key type .llcrypt unregistered [13301.835856] Key type ._llcrypt unregistered [13337.433523] Key type ._llcrypt registered [13337.438111] Key type .llcrypt registered [13337.588850] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [13337.915677] Key type .llcrypt unregistered [13337.919456] Key type ._llcrypt unregistered [13408.504753] Key type ._llcrypt registered [13408.506289] Key type .llcrypt registered [13408.720455] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [13409.141605] Key type .llcrypt unregistered [13409.143292] Key type ._llcrypt unregistered [13446.996647] Key type ._llcrypt registered [13446.998021] Key type .llcrypt registered [13447.084022] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [13447.324661] Key type .llcrypt unregistered [13447.326699] Key type ._llcrypt unregistered [13481.518270] Key type ._llcrypt registered [13481.519784] Key type .llcrypt registered [13481.616103] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [13481.872412] Key type .llcrypt unregistered [13481.875247] Key type ._llcrypt unregistered [13541.557265] Key type ._llcrypt registered [13541.559884] Key type .llcrypt registered [13541.753447] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [13559.480166] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [13560.563420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13560.632069] alg: No test for adler32 (adler32-zlib) [13562.073523] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [13562.303322] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [13563.967894] Key type lgssc registered [13565.547851] Lustre: Echo OBD driver; http://www.lustre.org/ [13602.659444] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13602.677547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13603.887363] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13603.915436] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13603.989659] Lustre: tst250fs-MDT0000: new disk, initializing [13604.076556] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13604.095141] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13608.309664] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [13619.982984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13620.096813] Lustre: 374650:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13620.154153] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13620.158140] Lustre: Skipped 1 previous similar message [13620.228183] Lustre: tst250fs-MDT0001: new disk, initializing [13620.374796] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13620.421354] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13620.435905] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13625.509753] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [13631.491220] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13638.842406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13639.182291] Lustre: tst250fs-OST0000: new disk, initializing [13639.186153] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13639.193967] Lustre: 376285:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13639.321799] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13642.639012] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13642.659514] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13642.796489] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13645.510479] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [13656.756021] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13656.962387] Lustre: tst250fs-OST0001: new disk, initializing [13656.966254] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13656.971054] Lustre: 377156:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13657.048544] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13664.281827] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [13665.927271] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13665.942992] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13666.086537] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13670.436543] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13672.551912] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13672.559547] Lustre: Skipped 1 previous similar message [13691.880097] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13691.885596] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13691.913865] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13696.992837] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13697.005905] Lustre: Skipped 3 previous similar messages [13702.127472] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13702.136586] Lustre: Skipped 4 previous similar messages [13706.207187] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13706.426276] Lustre: server umount tst250fs-MDT0000 complete [13710.453477] LustreError: 374643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788677845 with bad export cookie 11496033700433543978 [13710.458787] LustreError: 374643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13710.458809] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13711.134360] Lustre: server umount tst250fs-MDT0001 complete [13725.705609] Lustre: server umount tst250fs-OST0000 complete [13728.735712] Lustre: 372875:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788677847/real 1788677847] req@ffff9f8646dbaa00 x1875564504882688/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788677863 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13728.764445] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13728.784353] Lustre: Skipped 3 previous similar messages [13730.741784] Lustre: server umount tst250fs-OST0001 complete [13752.095303] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [13760.895777] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [13793.370514] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 02:58:47 (1788677927) [13795.340768] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13817.409988] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [13831.044429] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [13863.271896] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13620 sec ========= 02:59:57 (1788677997) [13864.814702] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:59:59 (1788677999) === [13868.852057] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:00:03 (1788678003) === [13908.247948] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [13910.740802] Key type lgssc unregistered [13911.025694] LNet: 386323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13911.036910] LNetError: 386323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13911.062283] LNet: Removed LNI 192.168.201.122@tcp [13911.972781] Key type .llcrypt unregistered [13911.974188] Key type ._llcrypt unregistered