[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 761603906 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003635] x2apic enabled [ 0.004012] Switched APIC routing to physical x2apic. [ 0.006005] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009035] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011019] pid_max: default: 32768 minimum: 301 [ 0.013161] LSM: Security Framework initializing [ 0.014084] Yama: becoming mindful. [ 0.016077] SELinux: Initializing. [ 0.018111] *** VALIDATE selinux *** [ 0.029148] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.034812] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.035190] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.036144] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.038018] *** VALIDATE tmpfs *** [ 0.040420] *** VALIDATE proc *** [ 0.042364] *** VALIDATE cgroup *** [ 0.043022] *** VALIDATE cgroup2 *** [ 0.045102] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.047133] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.048016] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.049045] Spectre V2 : User space: Vulnerable [ 0.050017] Speculative Store Bypass: Vulnerable [ 0.053834] debug: unmapping init [mem 0xffffffffb8259000-0xffffffffb8260fff] [ 0.056000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.058177] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.059035] ... version: 2 [ 0.060025] ... bit width: 48 [ 0.061020] ... generic registers: 4 [ 0.062027] ... value mask: 0000ffffffffffff [ 0.063034] ... max period: 00007fffffffffff [ 0.064025] ... fixed-purpose events: 3 [ 0.065027] ... event mask: 000000070000000f [ 0.066377] rcu: Hierarchical SRCU implementation. [ 0.070271] smp: Bringing up secondary CPUs ... [ 0.072699] x86: Booting SMP configuration: [ 0.073030] .... node #0, CPUs: #1 #2 #3 [ 0.085019] smp: Brought up 1 node, 4 CPUs [ 0.087025] smpboot: Max logical packages: 1 [ 0.088035] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.202036] node 0 deferred pages initialised in 111ms [ 0.209014] devtmpfs: initialized [ 0.210320] x86/mm: Memory block size: 128MB [ 0.213526] gcov: version magic: 0x41383552 [ 0.215229] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.224196] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.227969] pinctrl core: initialized pinctrl subsystem [ 0.231145] [ 0.231918] ************************************************************* [ 0.236017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.238018] ** ** [ 0.243020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.245020] ** ** [ 0.247017] ** This means that this kernel is built to expose internal ** [ 0.252020] ** IOMMU data structures, which may compromise security on ** [ 0.256016] ** your system. ** [ 0.260085] ** ** [ 0.262015] ** If you see this message and you are not debugging the ** [ 0.265020] ** kernel, report this immediately to your vendor! ** [ 0.267018] ** ** [ 0.270021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.273030] ************************************************************* [ 0.276649] NET: Registered protocol family 16 [ 0.279803] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.284091] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.288165] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.293116] cpuidle: using governor menu [ 0.297122] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.301499] PCI: Using configuration type 1 for base access [ 0.302000] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.313130] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.314112] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.315748] cryptd: max_cpu_qlen set to 1000 [ 0.318411] ACPI: Added _OSI(Module Device) [ 0.319018] ACPI: Added _OSI(Processor Device) [ 0.321090] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.323019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.329919] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.342704] ACPI: Interpreter enabled [ 0.346156] ACPI: PM: (supports S0 S3 S4 S5) [ 0.352096] ACPI: Using IOAPIC for interrupt routing [ 0.356140] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.361856] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.378062] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.381063] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.385024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.389100] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.396861] acpiphp: Slot [2] registered [ 0.399167] acpiphp: Slot [5] registered [ 0.401261] acpiphp: Slot [6] registered [ 0.403186] acpiphp: Slot [7] registered [ 0.405208] acpiphp: Slot [8] registered [ 0.407206] acpiphp: Slot [9] registered [ 0.409205] acpiphp: Slot [10] registered [ 0.411243] acpiphp: Slot [3] registered [ 0.412138] acpiphp: Slot [4] registered [ 0.413123] acpiphp: Slot [11] registered [ 0.415114] acpiphp: Slot [12] registered [ 0.417165] acpiphp: Slot [13] registered [ 0.418200] acpiphp: Slot [14] registered [ 0.421275] acpiphp: Slot [15] registered [ 0.423313] acpiphp: Slot [16] registered [ 0.426162] acpiphp: Slot [17] registered [ 0.429165] acpiphp: Slot [18] registered [ 0.431214] acpiphp: Slot [19] registered [ 0.434176] acpiphp: Slot [20] registered [ 0.436136] acpiphp: Slot [21] registered [ 0.438151] acpiphp: Slot [22] registered [ 0.440217] acpiphp: Slot [23] registered [ 0.442254] acpiphp: Slot [24] registered [ 0.444127] acpiphp: Slot [25] registered [ 0.446201] acpiphp: Slot [26] registered [ 0.449214] acpiphp: Slot [27] registered [ 0.451296] acpiphp: Slot [28] registered [ 0.453244] acpiphp: Slot [29] registered [ 0.455214] acpiphp: Slot [30] registered [ 0.458217] acpiphp: Slot [31] registered [ 0.460203] PCI host bridge to bus 0000:00 [ 0.462120] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.464039] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.467033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.470033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.474034] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.477087] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.479283] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.483550] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.488036] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.504064] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.510000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.512034] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.513225] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.517024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.519657] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.524108] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.530057] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.535032] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.551022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.567021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.574017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.582807] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.598019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.607023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.650021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.662353] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.678027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.690021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.723023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.747773] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.765016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.778017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.801026] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.825993] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.835017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.846019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.881021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.902207] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.907018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.920018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.954022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.964281] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.972015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.981016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.997000] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.015000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.017502] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.021606] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.025018] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.029247] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.035040] iommu: Default domain type: Passthrough [ 1.039166] SCSI subsystem initialized [ 1.042298] ACPI: bus type USB registered [ 1.045254] usbcore: registered new interface driver usbfs [ 1.049107] usbcore: registered new interface driver hub [ 1.052116] usbcore: registered new device driver usb [ 1.054192] pps_core: LinuxPPS API ver. 1 registered [ 1.056008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.061065] PTP clock support registered [ 1.064072] EDAC MC: Ver: 3.0.0 [ 1.071148] PCI: Using ACPI for IRQ routing [ 1.075592] NetLabel: Initializing [ 1.079012] NetLabel: domain hash size = 128 [ 1.081025] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.087083] NetLabel: unlabeled traffic allowed by default [ 1.092133] vgaarb: loaded [ 1.104325] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.109013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.117381] clocksource: Switched to clocksource kvm-clock [ 1.347771] VFS: Disk quotas dquot_6.6.0 [ 1.351948] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.354960] *** VALIDATE ramfs *** [ 1.356422] *** VALIDATE hugetlbfs *** [ 1.361656] pnp: PnP ACPI init [ 1.367078] pnp: PnP ACPI: found 6 devices [ 1.419556] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.424946] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.427515] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.430711] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.434283] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.444577] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.452563] NET: Registered protocol family 2 [ 1.463272] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.477345] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.484952] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.500501] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.511475] TCP: Hash tables configured (established 65536 bind 65536) [ 1.520396] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.529490] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.536740] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.539562] NET: Registered protocol family 1 [ 1.545847] RPC: Registered named UNIX socket transport module. [ 1.548125] RPC: Registered udp transport module. [ 1.555509] RPC: Registered tcp transport module. [ 1.561031] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.567233] NET: Registered protocol family 44 [ 1.569616] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.574791] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.580815] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.587429] PCI: CLS 0 bytes, default 64 [ 1.591446] Unpacking initramfs... [ 4.641083] debug: unmapping init [mem 0xffff9399fcc54000-0xffff9399fffbffff] [ 4.652183] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.661203] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.673309] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.440817] Initialise system trusted keyrings [ 6.443716] Key type blacklist registered [ 6.446283] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.464799] zbud: loaded [ 6.469585] *** VALIDATE nfs *** [ 6.474876] *** VALIDATE nfs4 *** [ 6.478844] pstore: using deflate compression [ 6.484490] Platform Keyring initialized [ 6.772606] NET: Registered protocol family 38 [ 6.777292] Key type asymmetric registered [ 6.779632] Asymmetric key parser 'x509' registered [ 6.786911] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.795646] io scheduler mq-deadline registered [ 6.803845] io scheduler kyber registered [ 6.811535] io scheduler bfq registered [ 6.819253] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.822481] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.837933] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.846257] ACPI: Power Button [PWRF] [ 6.862409] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.893387] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.941763] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.960775] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.994038] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.026358] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.055236] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.060940] Non-volatile memory driver v1.3 [ 7.063388] Linux agpgart interface v0.103 [ 7.108195] virtio_blk virtio1: [vda] 149984 512-byte logical blocks (76.8 MB/73.2 MiB) [ 7.117515] vda: detected capacity change from 0 to 76791808 [ 7.152792] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.158143] vdb: detected capacity change from 0 to 1073741824 [ 7.195474] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.208879] vdc: detected capacity change from 0 to 2621440000 [ 7.257046] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.266382] vdd: detected capacity change from 0 to 2621440000 [ 7.302940] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.313327] vde: detected capacity change from 0 to 4294967296 [ 7.349072] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.358817] vdf: detected capacity change from 0 to 4294967296 [ 7.381810] libphy: Fixed MDIO Bus: probed [ 7.390916] usbcore: registered new interface driver usbserial_generic [ 7.393829] usbserial: USB Serial support registered for generic [ 7.396330] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.401519] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.403604] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.406814] mousedev: PS/2 mouse device common for all mice [ 7.411120] rtc_cmos 00:05: RTC can wake from S4 [ 7.418767] rtc_cmos 00:05: registered as rtc0 [ 7.421345] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.421922] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.427124] intel_pstate: CPU model not supported [ 7.435372] hid: raw HID events driver (C) Jiri Kosina [ 7.440212] usbcore: registered new interface driver usbhid [ 7.443806] usbhid: USB HID core driver [ 7.445171] drop_monitor: Initializing network drop monitor service [ 7.448223] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.452669] Initializing XFRM netlink socket [ 7.458490] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.459823] NET: Registered protocol family 10 [ 7.472674] Segment Routing with IPv6 [ 7.474537] NET: Registered protocol family 17 [ 7.478073] mpls_gso: MPLS GSO support [ 7.486792] RAS: Correctable Errors collector initialized. [ 7.489521] AVX version of gcm_enc/dec engaged. [ 7.494285] AES CTR mode by8 optimization enabled [ 7.635047] sched_clock: Marking stable (7635024554, 0)->(9820488925, -2185464371) [ 7.648857] registered taskstats version 1 [ 7.664739] Loading compiled-in X.509 certificates [ 7.667946] zswap: loaded using pool lzo/zbud [ 7.759141] Key type big_key registered [ 7.777411] Key type encrypted registered [ 7.780378] ima: No TPM chip found, activating TPM-bypass! [ 7.783730] ima: Allocated hash algorithm: sha1 [ 7.788611] ima: No architecture policies found [ 7.797242] evm: Initialising EVM extended attributes: [ 7.803319] evm: security.selinux [ 7.810856] evm: security.ima [ 7.817202] evm: security.capability [ 7.821569] evm: HMAC attrs: 0x1 [ 7.826887] rtc_cmos 00:05: setting system clock to 2026-09-16 21:21:40 UTC (1789593700) [ 7.843569] debug: unmapping init [mem 0xffffffffb9203000-0xffffffffb93fffff] [ 7.849788] debug: unmapping init [mem 0xffffffffb7f82000-0xffffffffb8258fff] [ 7.858187] Write protecting the kernel read-only data: 28672k [ 7.862919] debug: unmapping init [mem 0xffffffffb6603000-0xffffffffb67fffff] [ 7.866417] debug: unmapping init [mem 0xffffffffb6f14000-0xffffffffb6ffffff] [ 7.960659] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.987729] systemd[1]: Detected virtualization kvm. [ 7.994711] systemd[1]: Detected architecture x86-64. [ 7.999690] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.079427] systemd[1]: No hostname configured. [ 8.082169] systemd[1]: Set hostname to . [ 8.097432] random: systemd: uninitialized urandom read (16 bytes read) [ 8.108636] systemd[1]: Initializing machine ID from random generator. [ 8.229579] random: ln: uninitialized urandom read (6 bytes read) [ 8.387695] random: systemd: uninitialized urandom read (16 bytes read) [ 8.394518] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 8.401392] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 8.413604] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.376357] device-mapper: uevent: version 1.0.3 [ 10.382739] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 12.840354] virtio_net virtio0 ens2: renamed from eth0 [ 12.899350] random: fast init done [ 13.486349] scsi host0: ata_piix [ 13.591692] scsi host1: ata_piix [ 13.597040] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 13.610197] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 18.346348] random: crng init done [ 18.350917] random: 7 urandom warning(s) missed due to ratelimiting [ 21.850472] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ 24.645754] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 29.725692] printk: systemd: 26 output lines suppressed due to ratelimiting [ 31.576310] SELinux: Disabled at runtime. [ 31.844404] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 31.903258] systemd[1]: Detected virtualization kvm. [ 31.905223] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 35.597880] systemd[1]: initrd-switch-root.service: Succeeded. [ 35.610572] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 35.625569] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 35.634542] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 35.646755] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 35.684552] systemd[1]: Starting Journal Service... Starting Journal Service... [ 35.717711] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ 36.059579] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 36.982915] systemd[1]: sysroot-var-lib-nfs-rpc_pipefs.mount: Succeeded. [ 36.997499] systemd[1]: sysroot.mount: Succeeded. [ 37.030511] systemd[1]: Started Journal Service. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 38.269775] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 39.109969] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 39.213081] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 39.796452] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 39.871643] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit)[ 44.897155] Key type dns_resolver registered [*** ] A start job is running for Configur…only root support (10s / no limit)[ 45.834882] NFS: Registering the id_resolver key type [ 45.837694] Key type id_resolver registered [ 45.839984] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg316-server login: [ 80.324101] hrtimer: interrupt took 5084322 ns [ 113.338976] libcfs: loading out-of-tree module taints kernel. [ 113.471627] Key type ._llcrypt registered [ 113.473705] Key type .llcrypt registered [ 113.612351] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 132.757708] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 134.333664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 134.350528] alg: No test for adler32 (adler32-zlib) [ 135.944553] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 137.170768] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 139.079670] Key type lgssc registered [ 141.488628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 162.602274] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 207.867235] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 220.662491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 220.700699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 221.946401] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 221.988835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 222.115236] Lustre: lustre-MDT0000: new disk, initializing [ 222.227198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 222.252470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 226.151480] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.251519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 240.406865] Lustre: 6523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 240.466651] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 240.473409] Lustre: Skipped 1 previous similar message [ 240.621914] Lustre: lustre-MDT0001: new disk, initializing [ 240.720169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 240.765251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 240.793194] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 245.062426] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.694407] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 259.553983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 259.914445] Lustre: lustre-OST0000: new disk, initializing [ 259.917884] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 259.931318] Lustre: 8459:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 260.003823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 262.764986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 262.775498] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 262.892481] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 266.663970] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 281.421700] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 281.626848] Lustre: lustre-OST0001: new disk, initializing [ 281.632446] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 281.644900] Lustre: 9530:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 281.734838] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 289.336137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 289.346645] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 289.398560] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 289.525345] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 303.941156] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 312.601774] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 319.715793] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing check_logdir /tmp/testlogs/ [ 324.987796] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing yml_node [ 330.611608] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 334.348786] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 338.029222] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 339.799232] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 16 17:27:11 EDT 2026 [ 358.039677] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 359.714162] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 376.299231] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 376.303769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 376.311241] Lustre: Skipped 1 previous similar message [ 376.332036] Lustre: Skipped 3 previous similar messages [ 381.414594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 381.423813] Lustre: Skipped 3 previous similar messages [ 381.749803] Lustre: server umount lustre-MDT0000 complete [ 385.341094] LustreError: 6515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789594078 with bad export cookie 17284631518712079677 [ 385.353750] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 385.357948] LustreError: 6515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 386.533292] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.544827] Lustre: Skipped 2 previous similar messages [ 386.549350] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 386.552593] Lustre: Skipped 1 previous similar message [ 390.569610] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 391.869355] Lustre: server umount lustre-MDT0001 complete [ 395.868964] Lustre: server umount lustre-OST0000 complete [ 399.952244] Lustre: server umount lustre-OST0001 complete [ 407.590643] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 420.524499] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 452.801043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 453.097865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 453.151593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 453.218491] Lustre: lustre-MDT0000: new disk, initializing [ 453.289615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 453.310203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 457.184767] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 466.551825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 466.667452] Lustre: 15085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 466.713825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 466.717901] Lustre: Skipped 1 previous similar message [ 466.788422] Lustre: lustre-MDT0001: new disk, initializing [ 466.867333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 466.907895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 466.921154] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 471.086952] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 479.954250] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 484.950898] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 491.493119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 491.690077] Lustre: lustre-OST0000: new disk, initializing [ 491.696766] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 491.702278] Lustre: 17054:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 491.778680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 492.901736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 492.924927] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 493.117683] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 499.327285] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 511.209662] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 516.101558] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 516.415420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 521.638688] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 521.879650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 528.873428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 528.875887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.915062] Lustre: Skipped 1 previous similar message [ 528.927446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 528.935752] Lustre: Skipped 1 previous similar message [ 530.197593] Lustre: server umount lustre-OST0000 complete [ 538.592700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 538.607243] LustreError: Skipped 1 previous similar message [ 538.615320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 538.635383] Lustre: Skipped 1 previous similar message [ 538.647918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 538.656516] Lustre: Skipped 1 previous similar message [ 540.439710] Lustre: server umount lustre-MDT0000 complete [ 544.049038] LustreError: 15079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789594236 with bad export cookie 17284631518712081210 [ 544.050248] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 544.056985] LustreError: 15079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 544.414363] Lustre: server umount lustre-MDT0001 complete [ 549.613641] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 17:30:41 (1789594241) [ 566.900165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 567.341550] LustreError: 19929:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.369494] LustreError: 19929:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 567.449341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 571.625251] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 572.910816] LustreError: 19930:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.016854] LustreError: 19929:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 579.676867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 584.335285] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 590.562206] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 595.398994] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 597.477059] Lustre: Failing over lustre-MDT0000 [ 597.684590] Lustre: server umount lustre-MDT0000 complete [ 600.545928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 600.554151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 600.564926] Lustre: Skipped 1 previous similar message [ 600.569405] LustreError: 19929:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 604.644499] LustreError: 19929:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 604.661352] LustreError: 19929:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 614.884238] LustreError: 20656:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 614.907856] LustreError: 20656:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 616.972612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 617.092507] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 617.540249] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 622.122600] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 622.568673] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 622.580960] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 622.604110] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 631.950915] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 637.920895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 637.927155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 637.938466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 637.938477] Lustre: Skipped 2 previous similar messages [ 637.980459] Lustre: Skipped 2 previous similar messages [ 642.364192] Lustre: server umount lustre-MDT0000 complete [ 643.046784] LustreError: 20656:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 643.077710] LustreError: 20656:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 646.318078] LustreError: 19910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789594338 with bad export cookie 17284631518712082624 [ 646.329155] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 646.723058] Lustre: server umount lustre-MDT0001 complete [ 655.273936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 655.811992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 655.818665] Lustre: Skipped 2 previous similar messages [ 660.603389] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 668.509133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.922765] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 680.434676] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 685.534981] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 687.397785] Lustre: Failing over lustre-MDT0000 [ 687.713654] Lustre: server umount lustre-MDT0000 complete [ 689.121867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 689.140290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 689.148324] LustreError: 23273:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 689.161160] LustreError: 23273:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 706.156415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 706.347186] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 706.571589] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 710.813238] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 711.652723] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 711.656232] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 711.671492] Lustre: Skipped 1 previous similar message [ 711.694545] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 718.777180] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 727.010082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 727.010980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 727.023105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 727.030077] Lustre: Skipped 2 previous similar messages [ 727.037611] Lustre: Skipped 1 previous similar message [ 728.274576] Lustre: server umount lustre-MDT0000 complete [ 731.835983] LustreError: 24841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789594424 with bad export cookie 17284631518712083562 [ 731.841103] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 731.846562] LustreError: 24841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 732.062219] Lustre: server umount lustre-MDT0001 complete [ 740.892512] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 17:33:52 (1789594432) [ 760.207301] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 771.391632] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 807.638353] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 819.145201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 819.416723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 819.447137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 819.570869] Lustre: lustre-MDT0000: new disk, initializing [ 819.735904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 819.747652] Lustre: Skipped 2 previous similar messages [ 819.766816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 824.907314] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 836.098035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 836.248558] Lustre: 29644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 836.487717] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 841.330030] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 850.687425] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 856.173672] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 862.686579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 862.976234] Lustre: 31611:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 864.713942] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 864.763155] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 864.857531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 864.877876] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 869.199968] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 879.970678] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 888.135547] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 908.719632] Lustre: server umount lustre-OST0000 complete [ 913.380249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 913.398448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 913.425140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 923.272237] Lustre: server umount lustre-MDT0000 complete [ 924.130450] LustreError: 29652:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 924.156151] LustreError: 29652:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 925.155396] Lustre: 3652:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789594601/real 1789594601] req@ffff9399436af480 x1876525139090432/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789594617 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 927.263157] Lustre: 3655:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789594603/real 1789594603] req@ffff9399436ac700 x1876525139090816/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789594619 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [ 930.082792] LustreError: 30450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789594622 with bad export cookie 17284631518712083961 [ 930.116734] LustreError: 30450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 930.121244] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 930.475700] Lustre: server umount lustre-MDT0001 complete [ 944.352625] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 947.880871] Key type lgssc unregistered [ 948.245968] LNet: 33592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.263529] LNetError: 33592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.287860] LNet: Removed LNI 192.168.203.116@tcp [ 949.479155] Key type .llcrypt unregistered [ 949.481803] Key type ._llcrypt unregistered [ 973.728509] Key type ._llcrypt registered [ 973.731490] Key type .llcrypt registered [ 973.816774] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 974.731348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 974.895792] alg: No test for adler32 (adler32-zlib) [ 975.894968] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 976.172968] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 977.871888] Key type lgssc registered [ 978.827265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 986.735276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 986.769744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 988.182771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 992.647894] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 997.355261] LustreError: 35002:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 997.388423] LustreError: 35002:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1001.066477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1001.561801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1005.930670] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1012.860692] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1018.173631] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1024.532267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1025.054313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1027.145735] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 1031.507514] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1039.015526] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1051.476163] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 17:39:02 (1789594742) [ 1053.314876] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1055.007737] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1061.369410] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 17:39:12 (1789594752) [ 1068.709362] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 17:39:20 (1789594760) [ 1069.729683] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1078.094256] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1078.103654] Lustre: Skipped 2 previous similar messages [ 1079.773962] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 17:39:31 (1789594771) [ 1083.056652] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1083.063090] Lustre: Skipped 3 previous similar messages [ 1100.762230] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1100.781362] Lustre: Skipped 6 previous similar messages [ 1103.386258] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 17:39:54 (1789594794) [ 1105.991779] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1109.224464] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1118.003131] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 17:40:09 (1789594809) [ 1119.750197] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1119.753698] Lustre: Skipped 3 previous similar messages [ 1132.385724] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 17:40:23 (1789594823) [ 1145.834157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1145.835101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1145.837231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1145.869027] Lustre: Skipped 1 previous similar message [ 1150.962063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1150.968739] Lustre: Skipped 2 previous similar messages [ 1151.745773] Lustre: server umount lustre-OST0000 complete [ 1156.063991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1156.065922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1156.081602] Lustre: Skipped 1 previous similar message [ 1156.093737] Lustre: Skipped 1 previous similar message [ 1161.184366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1161.195750] Lustre: Skipped 1 previous similar message [ 1161.409759] Lustre: server umount lustre-MDT0000 complete [ 1165.285760] LustreError: 34981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789594857 with bad export cookie 17401192572634096358 [ 1165.286986] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1165.296408] LustreError: 34981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1165.650782] Lustre: server umount lustre-MDT0001 complete [ 1176.624690] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1179.590970] Key type lgssc unregistered [ 1180.026159] LNet: 42855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1180.034798] LNetError: 42855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1180.061934] LNet: Removed LNI 192.168.203.116@tcp [ 1181.055063] Key type .llcrypt unregistered [ 1181.061713] Key type ._llcrypt unregistered [ 1200.509980] Key type ._llcrypt registered [ 1200.512437] Key type .llcrypt registered [ 1200.664983] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 1218.424397] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1219.609620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1219.646895] alg: No test for adler32 (adler32-zlib) [ 1220.747704] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 1221.002294] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1222.679195] Key type lgssc registered [ 1223.974281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1261.857270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1261.901597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1263.263938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1263.293047] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1263.387710] Lustre: lustre-MDT0000: new disk, initializing [ 1263.478292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1263.492692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1268.619797] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1280.183479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1280.325385] Lustre: 46009:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1280.420548] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1280.428892] Lustre: Skipped 1 previous similar message [ 1280.559184] Lustre: lustre-MDT0001: new disk, initializing [ 1280.660712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1280.716670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1280.741980] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1285.635276] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1296.953915] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1303.191829] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1309.877415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1310.120545] Lustre: lustre-OST0000: new disk, initializing [ 1310.125244] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1310.131593] Lustre: 47977:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1310.224679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1316.320845] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1316.453577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1316.477271] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1316.555719] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1325.932737] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1330.931640] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1331.096569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1335.595337] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1335.892897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1342.432524] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1342.442749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1342.461541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1342.463799] Lustre: Skipped 1 previous similar message [ 1342.482281] Lustre: Skipped 1 previous similar message [ 1343.893966] Lustre: server umount lustre-OST0000 complete [ 1347.552948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1347.561249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1347.562806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1347.562813] LustreError: Skipped 1 previous similar message [ 1347.565677] Lustre: Skipped 1 previous similar message [ 1347.605843] Lustre: Skipped 1 previous similar message [ 1352.678822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1357.794209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1357.803331] Lustre: Skipped 2 previous similar messages [ 1361.375113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1361.833864] Lustre: server umount lustre-MDT0000 complete [ 1366.025335] LustreError: 46002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789595058 with bad export cookie 15379192035168904000 [ 1366.030785] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1366.037736] LustreError: 46002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1366.341213] Lustre: server umount lustre-MDT0001 complete [ 1376.957497] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 17:44:28 (1789595068) [ 1384.212349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1384.790788] LustreError: 50439:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1384.818038] LustreError: 50439:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1384.887417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1388.553674] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1390.055047] LustreError: 50440:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1395.180442] LustreError: 50439:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1396.843152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1397.243516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1402.388882] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1410.857143] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1416.961323] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1424.129173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1424.488414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1431.100258] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1438.793802] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1446.580212] LustreError: 52968:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1447.517559] Lustre: Modifying parameter general.timeout=1 in log params [ 1449.324559] Lustre: Modifying parameter general.timeout=3 in log params [ 1449.329152] Lustre: Skipped 1 previous similar message [ 1451.973603] Lustre: Modifying parameter general.timeout=6 in log params [ 1451.979161] Lustre: Skipped 2 previous similar messages [ 1456.150232] Lustre: Modifying parameter general.timeout=11 in log params [ 1456.154130] Lustre: Skipped 4 previous similar messages [ 1464.803500] Lustre: Modifying parameter general.timeout=22 in log params [ 1464.808135] Lustre: Skipped 10 previous similar messages [ 1481.637792] Lustre: Modifying parameter general.timeout=40 in log params [ 1481.643025] Lustre: Skipped 17 previous similar messages [ 1514.004534] Lustre: Modifying parameter general.timeout=83 in log params [ 1514.010974] Lustre: Skipped 42 previous similar messages [ 1567.394174] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 17:47:38 (1789595258) [ 1570.000411] Lustre: 59905:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1571.960513] Lustre: 60003:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1575.309509] Lustre: 60216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1575.319995] Lustre: 60216:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1580.785156] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule= in log params [ 1580.789926] Lustre: Skipped 54 previous similar messages [ 1609.374894] Lustre: 61775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1609.386720] Lustre: 61775:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1609.409814] LustreError: 61775:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1615.585734] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 17:48:25 (1789595305) [ 1632.736257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1632.748519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1632.769918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1636.320522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1636.346935] Lustre: Skipped 1 previous similar message [ 1638.958899] Lustre: server umount lustre-MDT0000 complete [ 1641.444936] LustreError: 50439:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1641.475255] LustreError: 50439:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1643.218969] LustreError: 50421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789595335 with bad export cookie 15379192035168904882 [ 1643.227243] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1643.232701] LustreError: 50421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1643.624771] Lustre: server umount lustre-MDT0001 complete [ 1657.919262] Lustre: server umount lustre-OST0000 complete [ 1685.841354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1686.212926] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1686.517136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1690.987271] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1702.110481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1702.212723] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1702.251272] Lustre: 64307:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1708.333391] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1716.866365] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1723.273528] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1729.986144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1730.112075] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1730.357697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1730.365806] Lustre: Skipped 1 previous similar message [ 1736.052724] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1743.108753] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1744.662915] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1744.669481] Lustre: Skipped 15 previous similar messages [ 1744.727851] Lustre: 66872:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1753.062790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1753.070389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1753.087156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1753.092908] Lustre: Skipped 3 previous similar messages [ 1757.292738] Lustre: server umount lustre-OST0000 complete [ 1763.809886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1763.823089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1763.827817] Lustre: Skipped 1 previous similar message [ 1763.844960] LustreError: Skipped 1 previous similar message [ 1767.453269] Lustre: server umount lustre-MDT0000 complete [ 1768.930249] LustreError: 64315:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1768.954266] LustreError: 64315:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1771.027065] LustreError: 64299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789595463 with bad export cookie 15379192035168907850 [ 1771.030657] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1771.040398] LustreError: 64299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1771.346423] Lustre: server umount lustre-MDT0001 complete [ 1781.078974] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1783.705289] Key type lgssc unregistered [ 1783.961090] LNet: 68081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.970407] LNetError: 68081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1785.006324] LNet: Removed LNI 192.168.203.116@tcp [ 1785.842602] Key type .llcrypt unregistered [ 1785.847767] Key type ._llcrypt unregistered [ 1814.496644] Key type ._llcrypt registered [ 1814.498980] Key type .llcrypt registered [ 1814.624424] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1815.977254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1816.002606] alg: No test for adler32 (adler32-zlib) [ 1817.113881] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 1817.448443] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1819.143206] Key type lgssc registered [ 1820.300864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1829.110929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1829.135522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1830.772618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1835.721632] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1841.122510] LustreError: 69496:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1841.142822] LustreError: 69496:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1845.223251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1846.144288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1851.179409] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1859.202852] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1864.592862] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1872.283700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1873.027147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1880.192861] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1889.499731] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1898.989789] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1911.266606] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1911.269459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1911.280876] Lustre: Skipped 2 previous similar messages [ 1911.290367] Lustre: Skipped 2 previous similar messages [ 1914.998792] Lustre: server umount lustre-MDT0000 complete [ 1916.385946] LustreError: 69496:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1916.412723] LustreError: 69496:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1918.894045] LustreError: 72184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789595611 with bad export cookie 5782190129217626931 [ 1918.902837] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1918.919801] LustreError: 72184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1919.201964] Lustre: server umount lustre-MDT0001 complete [ 1932.641525] Lustre: server umount lustre-OST0000 complete [ 1960.623805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1960.861511] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1960.886376] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1961.145084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1965.722515] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1974.692741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1974.766112] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1974.808410] Lustre: 74372:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1975.419457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1980.106490] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1987.563980] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1993.069225] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1999.979657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2000.213544] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2000.797857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2007.708515] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2015.247761] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2016.824723] Lustre: 76946:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2018.639617] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2026.976778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2026.986221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2026.994364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2032.611508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2032.617324] Lustre: Skipped 2 previous similar messages [ 2037.731073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2037.746796] Lustre: Skipped 1 previous similar message [ 2038.239152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2038.483070] Lustre: server umount lustre-OST0000 complete [ 2041.825481] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2041.844699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2041.862599] Lustre: Skipped 1 previous similar message [ 2041.868443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2041.870254] Lustre: Skipped 1 previous similar message [ 2053.092178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2053.106625] Lustre: Skipped 5 previous similar messages [ 2056.163652] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2056.324306] Lustre: server umount lustre-MDT0000 complete [ 2058.210893] LustreError: 76499:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2058.236450] LustreError: 76499:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2060.534234] LustreError: 74362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789595753 with bad export cookie 5782190129217627778 [ 2060.545126] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2060.948664] Lustre: server umount lustre-MDT0001 complete [ 2075.717062] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2079.406154] Key type lgssc unregistered [ 2079.651265] LNet: 78156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2079.661213] LNetError: 78156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2079.676821] LNet: Removed LNI 192.168.203.116@tcp [ 2080.577168] Key type .llcrypt unregistered [ 2080.580721] Key type ._llcrypt unregistered [ 2107.435744] Key type ._llcrypt registered [ 2107.437533] Key type .llcrypt registered [ 2107.560715] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2108.874221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2108.909739] alg: No test for adler32 (adler32-zlib) [ 2110.150349] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 2110.478556] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2112.215274] Key type lgssc registered [ 2113.704863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2121.722462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2121.787854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2123.405040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2127.874664] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2133.988637] LustreError: 79566:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2134.000145] LustreError: 79566:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2136.382800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2137.137749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2141.678906] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2148.467886] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2154.033937] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2160.446322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2161.034467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2167.372101] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2174.414495] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2181.007180] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2181.899713] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2183.522537] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2183.532508] Lustre: Skipped 1 previous similar message [ 2185.664422] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2185.672801] Lustre: Skipped 2 previous similar messages [ 2190.356531] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2190.363427] Lustre: Skipped 5 previous similar messages [ 2199.115304] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2199.120304] Lustre: Skipped 8 previous similar messages [ 2215.357904] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2215.368056] Lustre: Skipped 16 previous similar messages [ 2247.498750] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2247.508173] Lustre: Skipped 31 previous similar messages [ 2284.188660] Lustre: 87670:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2287.310829] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 17:59:38 (1789595978) [ 2288.758602] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2290.551584] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 17:59:41 (1789595981) [ 2295.268122] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2295.280886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2295.300859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2298.337583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2298.343618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2298.344490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2298.345671] Lustre: Skipped 1 previous similar message [ 2300.690217] Lustre: server umount lustre-OST0000 complete [ 2306.021133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2306.033376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2306.044392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2308.576448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2308.578376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2308.623841] Lustre: Skipped 1 previous similar message [ 2310.417881] Lustre: server umount lustre-MDT0000 complete [ 2313.696348] LustreError: 84349:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2313.713574] LustreError: 84349:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2314.104449] LustreError: 80306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789596006 with bad export cookie 4903394804069522346 [ 2314.117908] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2314.568371] Lustre: server umount lustre-MDT0001 complete [ 2325.455740] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2328.204538] Key type lgssc unregistered [ 2328.593907] LNet: 88918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2328.603481] LNetError: 88918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2328.637122] LNet: Removed LNI 192.168.203.116@tcp [ 2329.586484] Key type .llcrypt unregistered [ 2329.590487] Key type ._llcrypt unregistered [ 2334.507462] Key type ._llcrypt registered [ 2334.516401] Key type .llcrypt registered [ 2334.632903] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2339.191650] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules [ 2340.369588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2340.528641] alg: No test for adler32 (adler32-zlib) [ 2341.597451] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 2341.897071] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2343.631304] Key type lgssc registered [ 2345.126086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2346.352132] LustreError: 90134:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2362.426924] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2367.529323] LustreError: 90134:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2372.998639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2373.033463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2374.744782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2379.335217] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2384.867911] LustreError: 91154:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2384.906358] LustreError: 91154:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2388.511087] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 18:01:19 (1789596079) [ 2389.986047] LustreError: 91155:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2393.988780] LustreError: 91186:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2394.000933] LustreError: 91186:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2394.007904] LustreError: 91186:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2394.372688] Lustre: server umount lustre-MDT0000 complete [ 2412.014951] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2414.548762] Key type lgssc unregistered [ 2414.837505] LNet: 92678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2414.849823] LNetError: 92678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2414.863155] LNet: Removed LNI 192.168.203.116@tcp [ 2415.799667] Key type .llcrypt unregistered [ 2415.803095] Key type ._llcrypt unregistered [ 2431.331732] Key type ._llcrypt registered [ 2431.335165] Key type .llcrypt registered [ 2431.457191] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2432.388687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2432.417333] alg: No test for adler32 (adler32-zlib) [ 2433.615326] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 2433.865259] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2435.591222] Key type lgssc registered [ 2436.713760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2444.932690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2444.963652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2446.519737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2450.903121] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2457.058467] LustreError: 93889:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2457.069196] LustreError: 93889:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2459.033228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2459.674611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2464.018437] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2471.120448] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2476.441771] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2483.809189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2484.401327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2490.928292] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2498.555436] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2545.054055] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 18:03:56 (1789596236) [ 2546.925397] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2548.695556] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 18:03:59 (1789596239) [ 2556.389930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2556.392513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2556.408606] Lustre: Skipped 2 previous similar messages [ 2556.422241] Lustre: Skipped 2 previous similar messages [ 2559.066673] Lustre: server umount lustre-MDT0000 complete [ 2561.504507] LustreError: 94910:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2561.529481] LustreError: 94910:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2562.522745] LustreError: 93871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789596255 with bad export cookie 16443106332598524610 [ 2562.532877] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2562.539014] LustreError: 93871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2562.828429] Lustre: server umount lustre-MDT0001 complete [ 2577.315226] Lustre: server umount lustre-OST0000 complete [ 2588.502929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2588.942654] LustreError: 98164:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.152428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2593.260551] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2594.279555] LustreError: 98165:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2594.301746] LustreError: 98165:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2598.379350] LustreError: 98164:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2602.176219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2603.021552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2607.976515] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2616.413157] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2623.105500] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2638.059355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2638.297772] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2638.305570] LustreError: 98154:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2638.346533] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2638.359748] LustreError: 100248:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2638.380803] LustreError: 100248:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2638.387287] LustreError: 100248:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2638.488356] Lustre: server umount lustre-OST0000 complete [ 2638.494252] LustreError: 100248:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2647.717068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2647.989100] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2647.991251] LustreError: 100719:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2647.995379] LustreError: 98154:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2648.037909] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2648.071890] LustreError: 100714:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2648.077964] LustreError: 100714:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2648.086812] LustreError: 100714:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2648.158763] Lustre: server umount lustre-OST0000 complete [ 2648.165956] LustreError: 100714:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2657.308294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2657.498176] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2657.502637] Lustre: Found index 0 for lustre-OST0000, updating log [ 2657.510975] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2657.574392] Lustre: lustre-OST0000: new disk, initializing [ 2657.581853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2657.589120] Lustre: 101248:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2657.877506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2663.960456] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2667.517950] Lustre: Failing over lustre-OST0000 [ 2667.812330] Lustre: server umount lustre-OST0000 complete [ 2674.144859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2674.160605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2674.190104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2677.729404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2677.737449] Lustre: Skipped 2 previous similar messages [ 2681.940613] Lustre: server umount lustre-MDT0000 complete [ 2682.854342] LustreError: 98895:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2682.867707] LustreError: 98895:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2685.577439] LustreError: 100268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789596378 with bad export cookie 16443106332598527354 [ 2685.585577] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2685.587981] LustreError: 100268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2685.902048] Lustre: server umount lustre-MDT0001 complete [ 2694.567901] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 18:06:25 (1789596385) [ 2710.897487] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2721.511434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2721.922536] LustreError: 103594:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2722.073864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2727.136984] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2735.862365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2742.019784] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2745.140840] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2752.827479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2753.532473] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2760.687886] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2760.706885] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2760.727765] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2761.511671] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2765.797189] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2765.813532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2766.848416] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:03 [ 2766.866905] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2766.871429] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2766.876972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2772.039929] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2772.159069] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2772.342883] Lustre: lustre-OST0001: new disk, initializing [ 2772.360719] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2772.383376] Lustre: 105915:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2772.959770] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2772.964760] Lustre: Skipped 2 previous similar messages [ 2778.810550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2778.822781] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2778.962257] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2781.260677] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2790.392923] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2800.602909] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2808.289302] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2808.297232] LustreError: Skipped 1 previous similar message [ 2808.313803] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2808.336728] Lustre: Skipped 1 previous similar message [ 2808.345730] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2811.472909] Lustre: server umount lustre-MDT0001 complete [ 2814.964075] LustreError: 105406:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2815.003669] LustreError: 105406:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2822.030043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2822.094921] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2822.097354] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2822.154416] Lustre: 103584:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2824.054248] LustreError: 107494:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2824.068969] LustreError: 107494:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2824.091210] LustreError: 103586:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2824.226587] LustreError: 107503:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.116@tcp: cfg command failed: rc = -17 [ 2824.234871] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2824.234871] [ 2827.552752] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2828.290745] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2838.387380] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 18:08:49 (1789596529) [ 2858.854404] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2881.237846] Lustre: 109532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2907.808835] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2912.572130] Lustre: 110668:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3001.824715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3001.840691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3001.858125] Lustre: Skipped 2 previous similar messages [ 3001.862722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3001.869140] Lustre: Skipped 4 previous similar messages [ 3005.161956] Lustre: server umount lustre-MDT0000 complete [ 3007.459841] LustreError: 103589:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3007.487069] LustreError: 103589:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 3009.209678] LustreError: 103576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789596701 with bad export cookie 16443106332598528355 [ 3009.213938] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3009.217988] LustreError: 103576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3009.582522] Lustre: server umount lustre-MDT0001 complete [ 3024.365885] Lustre: server umount lustre-OST0000 complete [ 3028.965289] Lustre: 93201:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789596705/real 1789596705] req@ffff93994f0e5180 x1876527547781760/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789596721 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3028.991467] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3029.001258] Lustre: Skipped 3 previous similar messages [ 3029.179961] Lustre: server umount lustre-OST0001 complete [ 3035.404895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3047.982926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3059.373905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3070.593878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3093.052765] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3105.326647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3105.356106] Lustre: lustre-MDT0000: reset Object Index mappings [ 3106.290198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3106.296641] Lustre: Skipped 1 previous similar message [ 3111.691380] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3122.800350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3122.838878] Lustre: lustre-MDT0001: reset Object Index mappings [ 3128.865885] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3133.213655] Lustre: 115770:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3141.232882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3148.086158] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3148.754069] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3151.152397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3157.996785] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3163.659312] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3163.661596] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3165.645852] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3173.246946] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3177.429761] Lustre: 117644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3227.631409] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 18:15:18 (1789596918) [ 3255.781467] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3255.799470] Lustre: Skipped 3 previous similar messages [ 3255.806451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3255.826975] Lustre: Skipped 7 previous similar messages [ 3259.628049] Lustre: server umount lustre-MDT0000 complete [ 3263.321200] LustreError: 114600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789596955 with bad export cookie 16443106332598575430 [ 3263.330331] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3263.341366] LustreError: 114600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3266.016444] LustreError: 114615:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3266.053339] LustreError: 114615:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [ 3269.864812] Lustre: server umount lustre-MDT0001 complete [ 3279.742743] Lustre: server umount lustre-OST0000 complete [ 3289.528546] Lustre: server umount lustre-OST0001 complete [ 3296.066777] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 3307.503765] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3344.015031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3344.247399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3344.273709] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3344.362401] Lustre: lustre-MDT0000: new disk, initializing [ 3344.429119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3344.434922] Lustre: Skipped 3 previous similar messages [ 3344.448116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3348.936940] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3356.293539] Lustre: server umount lustre-MDT0000 complete [ 3365.058482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3365.567995] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3365.578951] Lustre: 121914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3365.591445] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3371.109520] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3376.809944] Lustre: server umount lustre-MDT0000 complete [ 3387.047375] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 18:17:58 (1789597078) [ 3388.427551] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3390.379985] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 18:18:01 (1789597081) [ 3410.132907] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3423.711251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3430.001847] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3440.024243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3440.208807] Lustre: 123965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3440.296997] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3440.303282] Lustre: Skipped 1 previous similar message [ 3440.392869] Lustre: lustre-MDT0001: new disk, initializing [ 3440.589749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3440.601917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3445.743653] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3450.777720] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3457.159785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3457.453162] Lustre: lustre-OST0000: new disk, initializing [ 3457.456736] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3457.462384] Lustre: 125544:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3459.559318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3459.579905] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3459.715438] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3464.854930] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3476.961923] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3477.292678] Lustre: lustre-OST0001: new disk, initializing [ 3477.295405] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3477.331294] Lustre: 126415:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3478.542605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3478.558939] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3478.696765] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3485.595526] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3497.478110] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3502.719439] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3901.275774] Lustre: 132839:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3901.286289] Lustre: 132839:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3901.294878] Lustre: 132839:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3901.307466] Lustre: 132839:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3901.317937] Lustre: 132839:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3901.330300] Lustre: 132839:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4159.968820] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 18:30:50 (1789597850) [ 4174.818502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4174.835414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4174.836261] Lustre: Skipped 5 previous similar messages [ 4174.860442] Lustre: Skipped 7 previous similar messages [ 4179.944230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4179.959393] Lustre: Skipped 3 previous similar messages [ 4180.843369] Lustre: server umount lustre-MDT0000 complete [ 4184.479128] LustreError: 123957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789597877 with bad export cookie 16443106332598643197 [ 4184.486566] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4184.502791] LustreError: 123957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4185.059325] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4185.062414] LustreError: 127223:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4185.062769] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4185.085070] Lustre: Skipped 1 previous similar message [ 4185.131058] LustreError: 127223:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4191.218206] Lustre: server umount lustre-MDT0001 complete [ 4201.398366] Lustre: server umount lustre-OST0000 complete [ 4211.793440] Lustre: server umount lustre-OST0001 complete [ 4218.531382] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 4228.142717] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4269.467391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4269.857963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4269.944075] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4270.130496] Lustre: lustre-MDT0000: new disk, initializing [ 4270.284437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4270.292926] Lustre: Skipped 5 previous similar messages [ 4270.345156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4276.501643] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4290.103393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4290.259221] Lustre: 135888:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4290.341270] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4290.344271] Lustre: Skipped 1 previous similar message [ 4290.400581] Lustre: lustre-MDT0001: new disk, initializing [ 4290.494419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4290.518298] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4295.660133] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4305.768871] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4311.806639] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4318.864367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4319.217860] Lustre: lustre-OST0000: new disk, initializing [ 4319.222987] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4319.238261] Lustre: 137858:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4319.395677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4319.406738] Lustre: Skipped 1 previous similar message [ 4321.183288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4321.220095] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4321.330420] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4326.730792] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4337.137714] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4348.686785] Lustre: lustre-MDD0000: changelog on [ 4352.010730] Lustre: lustre-MDD0001: changelog on [ 6140.996278] Lustre: lustre-MDD0001: changelog off [ 6142.928201] Lustre: 139487:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6143.874678] Lustre: lustre-MDD0000: changelog off [ 6149.097663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6149.098215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6149.124647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6149.131589] Lustre: Skipped 3 previous similar messages [ 6154.209424] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6154.213587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6154.220666] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6155.407150] Lustre: server umount lustre-OST0000 complete [ 6159.333113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6159.334143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6159.335543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6159.335550] Lustre: Skipped 1 previous similar message [ 6159.347564] Lustre: Skipped 1 previous similar message [ 6164.449828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6164.461681] Lustre: Skipped 2 previous similar messages [ 6165.232433] Lustre: server umount lustre-MDT0000 complete [ 6168.232471] LustreError: 135880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789599860 with bad export cookie 16443106332599417796 [ 6168.234799] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6168.240154] LustreError: 135880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6168.573904] Lustre: server umount lustre-MDT0001 complete [ 6177.138166] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 6179.671664] Key type lgssc unregistered [ 6179.947914] LNet: 146982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6179.963083] LNetError: 146982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6179.983594] LNet: Removed LNI 192.168.203.116@tcp [ 6180.882158] Key type .llcrypt unregistered [ 6180.887208] Key type ._llcrypt unregistered [ 6199.468800] Key type ._llcrypt registered [ 6199.471704] Key type .llcrypt registered [ 6199.574057] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 6214.287692] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6215.219638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6215.310820] alg: No test for adler32 (adler32-zlib) [ 6216.395302] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 6216.689196] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 6218.456151] Key type lgssc registered [ 6219.516841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6251.915387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6251.960538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6253.285210] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6253.317797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6253.440886] Lustre: lustre-MDT0000: new disk, initializing [ 6253.583317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6253.617486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6258.482522] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6272.097627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6272.267246] Lustre: 150180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6272.353506] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6272.358303] Lustre: Skipped 1 previous similar message [ 6272.466705] Lustre: lustre-MDT0001: new disk, initializing [ 6272.596112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6272.649235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6272.674671] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6277.755835] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6287.572396] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6292.795456] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6298.408759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6298.647791] Lustre: lustre-OST0000: new disk, initializing [ 6298.652250] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6298.657659] Lustre: 152146:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6298.751276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6304.043746] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6305.875938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6305.886847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6306.008205] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6313.696991] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6319.432685] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_module obdecho/obdecho [ 6321.153759] LustreError: 153150:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6321.164702] LustreError: 153150:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6322.771289] LustreError: 153247:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6333.924609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6333.931641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6333.956238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6336.994856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6336.997027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6337.014021] Lustre: Skipped 1 previous similar message [ 6337.014688] Lustre: Skipped 1 previous similar message [ 6339.275872] Lustre: server umount lustre-MDT0000 complete [ 6343.196885] LustreError: 150173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789600035 with bad export cookie 4464326469294480166 [ 6343.208829] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6343.218315] LustreError: 150173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6343.579947] Lustre: server umount lustre-MDT0001 complete [ 6357.865228] Lustre: server umount lustre-OST0000 complete [ 6380.129361] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 6390.783538] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6425.534191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6425.839370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6425.876781] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6425.992883] Lustre: lustre-MDT0000: new disk, initializing [ 6426.167805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6426.188041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6430.789539] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6441.635218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6441.729213] Lustre: 156966:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6441.785648] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6441.789748] Lustre: Skipped 1 previous similar message [ 6441.865234] Lustre: lustre-MDT0001: new disk, initializing [ 6441.955253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6442.004043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6442.024154] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6447.296488] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6456.606816] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6462.311458] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6469.316809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6469.575719] Lustre: lustre-OST0000: new disk, initializing [ 6469.579385] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6469.588430] Lustre: 158933:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6469.680894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6476.725563] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6476.858041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6476.884087] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6477.034188] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6486.129593] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6491.180553] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6491.440878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6496.635873] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6496.993223] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6502.881211] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6502.883185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6502.892762] LustreError: Skipped 1 previous similar message [ 6502.895399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6502.909993] Lustre: Skipped 1 previous similar message [ 6502.931343] Lustre: Skipped 2 previous similar messages [ 6505.486396] Lustre: server umount lustre-OST0000 complete [ 6513.120605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6513.126272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6513.147523] Lustre: Skipped 1 previous similar message [ 6513.180428] Lustre: Skipped 1 previous similar message [ 6516.226851] Lustre: server umount lustre-MDT0000 complete [ 6518.246507] LustreError: 156971:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6518.293649] LustreError: 156971:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6520.054935] LustreError: 156956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789600212 with bad export cookie 4464326469294481125 [ 6520.055639] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6520.069960] LustreError: 156956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6520.514053] Lustre: server umount lustre-MDT0001 complete [ 6524.374719] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 19:10:15 (1789600215) [ 6526.035128] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6527.982369] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 19:10:19 (1789600219) [ 6535.119170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6535.521121] LustreError: 161348:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6535.591176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6540.120339] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6540.771358] LustreError: 161349:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6545.895131] LustreError: 161348:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6548.968798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6554.945255] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6564.367270] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6571.408821] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6578.196564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6578.641963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6578.646380] Lustre: Skipped 1 previous similar message [ 6584.906301] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6592.463977] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6603.173839] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 6605.286571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6605.294467] LustreError: 161345:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6605.301181] Lustre: Skipped 1 previous similar message [ 6605.337347] LustreError: 161345:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6606.050758] Lustre: server umount lustre-MDT0000 complete [ 6609.824397] LustreError: 161328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789600302 with bad export cookie 4464326469294482007 [ 6609.840133] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6609.840427] LustreError: 161328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6613.385213] LustreError: 161343:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6613.409690] LustreError: 161343:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6613.412359] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.16@tcp (stopping) [ 6613.439931] Lustre: Skipped 1 previous similar message [ 6624.223146] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6624.528184] Lustre: server umount lustre-MDT0001 complete [ 6631.468518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6643.363464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6643.655410] LustreError: 165194:0:(llog.c:1655:llog_backup()) MGC192.168.203.116@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6643.666791] Lustre: 165194:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.116@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6650.242868] LustreError: 165198:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6650.255886] LustreError: 165198:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6684.744862] LustreError: 165200:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6684.771819] LustreError: 165200:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 28 previous similar messages [ 6704.775369] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6704.780375] Lustre: 165194:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.116@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6745.597853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6749.940609] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6750.609957] LustreError: 165200:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6750.659150] LustreError: 165200:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 40 previous similar messages [ 6750.697719] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6759.410735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6775.267582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6784.152443] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6785.030996] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6792.933397] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6794.573501] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6800.847448] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6802.476395] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6820.833023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6820.838435] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6820.844058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6820.848275] Lustre: Skipped 3 previous similar messages [ 6820.853199] Lustre: Skipped 4 previous similar messages [ 6826.587061] Lustre: server umount lustre-MDT0000 complete [ 6830.831050] LustreError: 161330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789600523 with bad export cookie 4464326469294488468 [ 6830.837450] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6830.841167] LustreError: 161330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6831.485844] Lustre: server umount lustre-MDT0001 complete [ 6837.464381] Lustre: server umount lustre-OST0000 complete [ 6862.242487] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 6874.249696] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6915.558616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6915.842920] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6915.887295] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6915.998353] Lustre: lustre-MDT0000: new disk, initializing [ 6916.102278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6916.112508] Lustre: Skipped 1 previous similar message [ 6916.132988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6920.796879] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6931.032411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6931.272904] Lustre: 170301:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6931.680370] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6936.573254] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6946.722348] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6952.428439] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6959.446042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6959.725882] Lustre: lustre-OST0000: new disk, initializing [ 6959.728838] Lustre: Skipped 1 previous similar message [ 6959.735306] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6959.741895] Lustre: Skipped 2 previous similar messages [ 6959.748557] Lustre: 172267:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6966.300679] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6967.899670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6967.907416] Lustre: Skipped 1 previous similar message [ 6967.915699] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6968.090011] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6976.254070] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6981.138689] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6981.377361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6986.825406] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6987.093816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6993.898030] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6993.904833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6993.911763] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6993.916363] Lustre: Skipped 3 previous similar messages [ 6993.922289] Lustre: Skipped 4 previous similar messages [ 6995.480972] Lustre: server umount lustre-OST0000 complete [ 7003.620433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7003.639725] LustreError: Skipped 1 previous similar message [ 7015.391359] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7015.708919] Lustre: server umount lustre-MDT0000 complete [ 7018.464783] LustreError: 170311:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7018.492809] LustreError: 170311:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 27 previous similar messages [ 7020.660351] LustreError: 170293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789600713 with bad export cookie 4464326469294505296 [ 7020.679207] LustreError: 170293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7020.696731] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7021.094983] Lustre: server umount lustre-MDT0001 complete [ 7031.474276] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 19:18:42 (1789600722) [ 7038.828188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7044.539916] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7053.985123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7058.895760] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7066.409960] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7073.627929] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7081.676535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7089.106389] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7097.174521] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7111.145041] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 19:20:01 (1789600801) [ 7118.816292] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7118.832178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7118.849734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7118.865070] Lustre: Skipped 3 previous similar messages [ 7118.870276] Lustre: Skipped 8 previous similar messages [ 7122.278091] Lustre: server umount lustre-OST0000 complete [ 7132.565721] Lustre: server umount lustre-MDT0000 complete [ 7136.091416] LustreError: 174663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789600828 with bad export cookie 4464326469294506178 [ 7136.099937] LustreError: 174663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7136.101384] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7136.549479] Lustre: server umount lustre-MDT0001 complete [ 7146.972206] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7149.775433] Key type lgssc unregistered [ 7150.027402] LNet: 178408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7150.042608] LNetError: 178408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7150.061685] LNet: Removed LNI 192.168.203.116@tcp [ 7150.860675] Key type .llcrypt unregistered [ 7150.864186] Key type ._llcrypt unregistered [ 7169.668650] Key type ._llcrypt registered [ 7169.672516] Key type .llcrypt registered [ 7169.788838] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7170.668712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7170.770715] alg: No test for adler32 (adler32-zlib) [ 7171.796388] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 7172.041269] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 7173.695199] Key type lgssc registered [ 7175.160880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7183.622903] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7183.650041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7199.263390] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7204.386076] LustreError: 179649:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.116@tcp: failed processing log, type 4: rc = -110 [ 7209.506548] LustreError: 179649:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7209.529945] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7209.539904] LustreError: MGC192.168.203.116@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7209.565768] LustreError: 179649:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7209.582992] LustreError: 179649:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 7209.593458] LustreError: 179649:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 7209.705254] Lustre: server umount lustre-OST0000 complete [ 7209.713742] LustreError: 179649:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7220.075091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7221.646182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7226.948713] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7231.969830] LustreError: 180148:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7231.990619] LustreError: 180148:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7236.533821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7237.061339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7242.661969] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7251.066345] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7257.270474] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7265.070329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7265.319885] LustreError: 181964:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7265.334245] Lustre: 181964:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.116@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7265.548815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7273.091249] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7281.457541] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7288.295325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7288.300510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7288.304419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7288.317859] Lustre: Skipped 2 previous similar messages [ 7288.347487] Lustre: Skipped 2 previous similar messages [ 7290.626403] Lustre: server umount lustre-MDT0000 complete [ 7293.409933] LustreError: 180143:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7293.445584] LustreError: 180143:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7295.176930] LustreError: 180129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789600987 with bad export cookie 5343552389625131879 [ 7295.187655] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7295.195382] LustreError: 180129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7295.796450] Lustre: server umount lustre-MDT0001 complete [ 7311.452313] Lustre: server umount lustre-OST0000 complete [ 7321.949906] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 7334.077757] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7375.463583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7375.723485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7375.759993] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7375.808978] Lustre: lustre-MDT0000: new disk, initializing [ 7375.910093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7375.947501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7380.945630] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7391.896585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7392.060924] Lustre: 185491:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7392.164218] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7392.175228] Lustre: Skipped 1 previous similar message [ 7392.308421] Lustre: lustre-MDT0001: new disk, initializing [ 7392.430236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7392.492213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7392.509499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7397.320768] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7409.211790] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7415.311547] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7422.563144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7422.853611] Lustre: lustre-OST0000: new disk, initializing [ 7422.866422] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7422.874952] Lustre: 187461:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7422.993492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7429.198279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7429.221866] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7429.404869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7430.021634] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7440.145706] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7445.810691] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7446.098366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7453.153886] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7453.524796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7460.322960] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7460.339112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7460.339899] LustreError: Skipped 1 previous similar message [ 7460.348838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7460.382745] Lustre: Skipped 1 previous similar message [ 7465.447482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7465.457780] Lustre: Skipped 2 previous similar messages [ 7470.047242] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7470.189850] Lustre: server umount lustre-OST0000 complete [ 7474.657829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7474.669395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7474.693270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7480.722451] Lustre: server umount lustre-MDT0000 complete [ 7480.800182] LustreError: 185504:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7480.830673] LustreError: 185504:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7484.612076] LustreError: 185483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789601177 with bad export cookie 5343552389625132600 [ 7484.632367] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7484.640517] LustreError: 185483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7485.182203] Lustre: server umount lustre-MDT0001 complete [ 7494.939995] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 19:26:26 (1789601186) [ 7511.124065] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7513.687624] Key type lgssc unregistered [ 7513.951900] LNet: 190408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7513.961104] LNetError: 190408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7513.974829] LNet: Removed LNI 192.168.203.116@tcp [ 7514.672951] Key type .llcrypt unregistered [ 7514.675864] Key type ._llcrypt unregistered [ 7534.006371] Key type ._llcrypt registered [ 7534.008857] Key type .llcrypt registered [ 7534.100432] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7535.123073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7535.209899] alg: No test for adler32 (adler32-zlib) [ 7536.293458] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 7536.585291] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 7538.271549] Key type lgssc registered [ 7539.494493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7547.555140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7547.579337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7549.276552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7553.881421] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7558.633330] LustreError: 191623:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7558.653266] LustreError: 191623:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7562.697576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7563.016693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7567.739703] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7576.671222] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7581.940379] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7590.509187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7590.780682] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7590.786850] LustreError: 193487:0:(llog.c:1696:llog_backup()) MGC192.168.203.116@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7590.792761] Lustre: 193487:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.116@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7590.940443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7596.802622] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7603.472944] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7607.459370] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7607.473916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7607.477804] Lustre: Skipped 1 previous similar message [ 7607.500996] Lustre: Skipped 2 previous similar messages [ 7611.727787] Lustre: server umount lustre-MDT0000 complete [ 7612.907142] LustreError: 191624:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7612.936740] LustreError: 191624:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7615.828213] LustreError: 193515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789601308 with bad export cookie 4673348179593493541 [ 7615.829961] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7615.840141] LustreError: 193515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7617.760378] LustreError: 192353:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000044cdb2c5 x1876532898327680/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7617.933375] Lustre: server umount lustre-MDT0001 complete [ 7632.158033] Lustre: server umount lustre-OST0000 complete [ 7642.596727] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 7653.342451] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7690.226435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7690.435038] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7690.508379] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7690.591835] Lustre: lustre-MDT0000: new disk, initializing [ 7690.688122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7690.710540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7695.813203] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7707.442891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7707.643657] Lustre: 197012:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7707.701515] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7707.703415] Lustre: Skipped 1 previous similar message [ 7707.706832] LustreError: 197794:0:(llog.c:1696:llog_backup()) MGC192.168.203.116@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7707.715778] LustreError: 197794:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7707.727640] Lustre: 197794:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.116@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7707.763433] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7707.769810] Lustre: Skipped 1 previous similar message [ 7707.918438] Lustre: lustre-MDT0001: new disk, initializing [ 7708.115254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7708.155836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7708.163955] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7712.959396] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7724.901854] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7734.632670] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7743.302216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7743.548888] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7743.550782] Lustre: Skipped 1 previous similar message [ 7743.554148] LustreError: 198957:0:(llog.c:1696:llog_backup()) MGC192.168.203.116@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7743.559877] LustreError: 198957:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7743.565193] Lustre: 198957:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.116@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7743.575758] Lustre: 198957:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7743.603751] Lustre: lustre-OST0000: new disk, initializing [ 7743.608982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7743.627908] Lustre: 198980:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7743.777615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7751.185213] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7752.301143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7752.317473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7752.494240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7762.009032] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7766.762189] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7767.023899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7771.303422] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7771.548472] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7778.273715] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7778.276491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7778.289652] LustreError: Skipped 1 previous similar message [ 7778.309283] Lustre: Skipped 1 previous similar message [ 7778.311689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7780.476673] Lustre: server umount lustre-OST0000 complete [ 7784.930619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7784.937431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7784.955661] Lustre: Skipped 1 previous similar message [ 7784.966694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7784.974081] Lustre: Skipped 1 previous similar message [ 7788.517583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7788.518487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7788.553767] Lustre: Skipped 1 previous similar message [ 7790.878099] Lustre: server umount lustre-MDT0000 complete [ 7793.635224] LustreError: 199570:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7793.650982] LustreError: 199570:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7794.838310] LustreError: 197806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789601487 with bad export cookie 4673348179593494262 [ 7794.843897] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7794.864669] LustreError: 197806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7795.242621] Lustre: server umount lustre-MDT0001 complete [ 7803.371692] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 19:31:34 (1789601494) [ 7821.617856] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7832.763234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7833.340922] LustreError: 202109:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7833.420217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7838.287276] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7838.703275] LustreError: 202110:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7846.930162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7853.338529] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7856.796200] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7864.113671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7872.549670] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7882.582049] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7882.724197] Lustre: lustre-OST0001: new disk, initializing [ 7882.737667] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7882.749939] Lustre: 204418:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7882.859481] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7882.870038] Lustre: Skipped 2 previous similar messages [ 7889.500318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7889.514608] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7889.644029] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7890.527940] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7903.496234] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7913.315328] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7916.046432] Lustre: DEBUG MARKER: ADD OST3 [ 7921.684636] loop: module loaded [ 7921.755788] loop0: detected capacity change from 0 to 204800000 [ 7921.969126] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7921.987528] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7922.011224] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7928.138779] loop0: detected capacity change from 0 to 204800000 [ 7929.275515] Lustre: DEBUG MARKER: STOP OST3 [ 7934.777357] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7934.920533] Lustre: lustre-OST0003: new disk, initializing [ 7934.928131] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7934.952519] Lustre: 206412:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7937.144174] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7937.285116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7937.308674] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7937.359811] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7939.167087] LustreError: 206602:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7939.600028] Lustre: Failing over lustre-OST0003 [ 7939.923938] Lustre: server umount lustre-OST0003 complete [ 7941.544045] LustreError: 203595:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7941.563026] LustreError: 203595:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7942.642315] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7942.643986] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7943.114106] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7952.177127] Lustre: DEBUG MARKER: START OST3 again [ 7958.005302] LustreError: 203595:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7958.025896] LustreError: 203595:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 7959.530656] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7959.849045] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7959.856493] Lustre: Skipped 1 previous similar message [ 7959.876478] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7961.378828] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7961.438828] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7961.439849] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7966.741598] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7987.595436] Lustre: Failing over lustre-OST0003 [ 7987.748394] Lustre: server umount lustre-OST0003 complete [ 7990.767414] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7990.777881] LustreError: 204727:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7990.795985] Lustre: Skipped 2 previous similar messages [ 7990.847703] LustreError: 204727:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8006.114522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8006.122348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8009.189854] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8009.203876] Lustre: Skipped 3 previous similar messages [ 8012.501416] Lustre: server umount lustre-MDT0000 complete [ 8016.708770] LustreError: 202092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789601709 with bad export cookie 4673348179593495137 [ 8016.710188] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8016.719701] LustreError: 202092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8019.458571] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8019.471611] Lustre: Skipped 5 previous similar messages [ 8031.200355] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8031.679442] Lustre: server umount lustre-MDT0001 complete [ 8035.627557] Lustre: server umount lustre-OST0000 complete [ 8039.369717] Lustre: server umount lustre-OST0001 complete [ 8046.506386] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 8054.433787] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8089.793648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8090.127063] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8090.196543] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8090.291893] Lustre: lustre-MDT0000: new disk, initializing [ 8090.420082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8090.446961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8095.460887] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8106.199881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8106.317154] Lustre: 211885:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8106.372897] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8106.381342] Lustre: Skipped 1 previous similar message [ 8106.488530] Lustre: lustre-MDT0001: new disk, initializing [ 8106.646602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8106.677994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8111.205122] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8122.465792] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8129.895525] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8138.103878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8138.520158] Lustre: lustre-OST0000: new disk, initializing [ 8138.526289] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8138.533657] Lustre: 213855:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8140.326056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8140.360200] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8140.531262] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8147.016742] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8158.196566] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8163.815751] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8164.087207] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8169.022298] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8169.225297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8171.496994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8171.500884] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8171.514376] Lustre: Skipped 2 previous similar messages [ 8171.516429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8171.534696] Lustre: Skipped 5 previous similar messages [ 8185.311141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8185.476817] Lustre: server umount lustre-OST0000 complete [ 8195.612616] Lustre: server umount lustre-MDT0000 complete [ 8197.090147] LustreError: 211898:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8197.106362] LustreError: 211898:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [ 8199.347996] LustreError: 211878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789601892 with bad export cookie 4673348179593497440 [ 8199.349091] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8199.358478] LustreError: 211878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8199.724553] Lustre: server umount lustre-MDT0001 complete [ 8202.982335] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 19:38:14 (1789601894) [ 8219.823319] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 8229.650416] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8272.825900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8273.219705] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8273.243241] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8273.330921] Lustre: lustre-MDT0000: new disk, initializing [ 8273.577329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8279.104938] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8290.464070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8290.597683] Lustre: 218467:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8290.878594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8295.940246] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8305.148889] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8311.174105] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8318.518810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8318.976177] Lustre: 220441:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8321.125386] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8326.480973] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8335.790638] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8340.172373] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8340.432358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8344.948848] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8345.162495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8351.712449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8351.719253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8351.725946] LustreError: Skipped 2 previous similar messages [ 8351.732553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8351.768360] Lustre: Skipped 4 previous similar messages [ 8351.787574] Lustre: Skipped 7 previous similar messages [ 8353.773800] Lustre: server umount lustre-OST0000 complete [ 8363.766298] Lustre: server umount lustre-MDT0000 complete [ 8367.072616] LustreError: 218480:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8367.113262] LustreError: 218480:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8367.777947] LustreError: 218459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602060 with bad export cookie 4673348179593498322 [ 8367.782212] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8367.811348] LustreError: 218459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8368.169207] Lustre: server umount lustre-MDT0001 complete [ 8378.083224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8378.739652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8378.750053] Lustre: Skipped 5 previous similar messages [ 8384.134457] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8394.761793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8400.942540] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8408.851965] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8415.451898] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8423.127660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8430.623169] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8438.730231] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8450.531301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8450.542256] Lustre: Skipped 4 previous similar messages [ 8455.317399] Lustre: server umount lustre-OST0000 complete [ 8465.769299] Lustre: server umount lustre-MDT0000 complete [ 8469.490903] LustreError: 222638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602162 with bad export cookie 4673348179593499197 [ 8469.496316] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8469.503393] LustreError: 222638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8469.862776] Lustre: server umount lustre-MDT0001 complete [ 8579.385724] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 8582.406246] Key type lgssc unregistered [ 8582.696130] LNet: 226317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8582.709671] LNetError: 226317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8582.737076] LNet: Removed LNI 192.168.203.116@tcp [ 8583.678170] Key type .llcrypt unregistered [ 8583.680362] Key type ._llcrypt unregistered [ 8607.956690] Key type ._llcrypt registered [ 8607.958769] Key type .llcrypt registered [ 8608.076370] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 8621.906724] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8622.677775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8622.693476] alg: No test for adler32 (adler32-zlib) [ 8623.701257] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 8623.920826] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 8625.615224] Key type lgssc registered [ 8626.730139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8663.423445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8663.476702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8664.890889] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8664.929085] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8665.083893] Lustre: lustre-MDT0000: new disk, initializing [ 8665.219628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8665.264205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8670.419791] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8682.842684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8682.983815] Lustre: 229667:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8683.042111] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8683.050690] Lustre: Skipped 1 previous similar message [ 8683.151125] Lustre: lustre-MDT0001: new disk, initializing [ 8683.310423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8683.390690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8683.410407] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8688.768882] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8698.312733] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8704.561956] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8711.653665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8711.936743] Lustre: lustre-OST0000: new disk, initializing [ 8711.941075] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8711.949402] Lustre: 231634:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8712.038158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8718.813831] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8721.956352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8721.965846] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8722.120688] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8727.506270] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8732.474877] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8732.652044] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8737.678991] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8737.886047] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8742.368434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8742.376347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8742.396086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8742.880612] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8742.897188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8742.908733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8748.013531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8753.122527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8753.134281] Lustre: Skipped 1 previous similar message [ 8754.144606] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8754.339666] Lustre: server umount lustre-OST0000 complete [ 8760.290929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8760.314238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8760.342665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8760.357976] Lustre: Skipped 1 previous similar message [ 8763.362655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8764.964977] Lustre: server umount lustre-MDT0000 complete [ 8769.127373] LustreError: 229658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602461 with bad export cookie 6524559626352120687 [ 8769.142911] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8769.144019] LustreError: 229658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8769.586221] Lustre: server umount lustre-MDT0001 complete [ 8789.646298] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8801.330429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8801.841260] LustreError: 234570:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8801.859958] LustreError: 234570:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8801.924232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8807.393102] LustreError: 234571:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8807.423132] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8811.492867] LustreError: 234570:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8816.616893] LustreError: 234571:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8818.540491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8819.121987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8824.164722] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8827.059338] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8833.873029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8834.263731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8840.345399] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8848.701389] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8848.989816] Lustre: lustre-OST0001: new disk, initializing [ 8848.996094] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8849.011584] Lustre: 236881:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8855.200145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8855.217086] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8855.376255] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8856.137823] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8868.112626] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8873.207478] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8886.437201] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 19:49:37 (1789602577) [ 8890.849068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8890.854857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8890.866963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8890.872199] Lustre: Skipped 2 previous similar messages [ 8895.148069] Lustre: server umount lustre-MDT0000 complete [ 8896.492291] LustreError: 234570:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8896.521636] LustreError: 234570:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8898.658971] LustreError: 234553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602591 with bad export cookie 6524559626352121569 [ 8898.660308] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8898.676504] LustreError: 234553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8899.038329] Lustre: server umount lustre-MDT0001 complete [ 8913.477729] Lustre: server umount lustre-OST0000 complete [ 8927.940395] Lustre: server umount lustre-OST0001 complete [ 8935.255543] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 8944.741468] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8985.626852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8985.854431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8985.883323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8985.962980] Lustre: lustre-MDT0000: new disk, initializing [ 8986.087607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8986.096904] Lustre: Skipped 1 previous similar message [ 8986.142738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8991.261033] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9003.223292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9003.378732] Lustre: 240830:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9003.450196] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9003.456952] Lustre: Skipped 1 previous similar message [ 9003.597414] Lustre: lustre-MDT0001: new disk, initializing [ 9003.778921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9003.796227] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9008.173645] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9017.277939] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9022.976624] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9029.686551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9030.021261] Lustre: lustre-OST0000: new disk, initializing [ 9030.026339] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9030.034905] Lustre: 242796:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9030.162294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9030.165723] Lustre: Skipped 1 previous similar message [ 9031.658749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9031.673606] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9031.761313] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9036.223918] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9045.490856] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9050.298122] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9050.516492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9054.823611] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9055.035580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9057.250613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9057.252411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9057.254176] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9057.262830] Lustre: Skipped 4 previous similar messages [ 9057.270333] Lustre: Skipped 4 previous similar messages [ 9063.123489] Lustre: server umount lustre-OST0000 complete [ 9081.311209] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9081.619156] Lustre: server umount lustre-MDT0000 complete [ 9082.856501] LustreError: 243385:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9082.872886] LustreError: 243385:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9085.391486] LustreError: 242795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602778 with bad export cookie 6524559626352123704 [ 9085.393061] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9085.409092] LustreError: 242795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9085.903492] Lustre: server umount lustre-MDT0001 complete [ 9094.821693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9095.300867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9099.644596] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9100.771988] LustreError: 245013:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9100.792859] LustreError: 245013:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9107.895920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9113.212570] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9120.569269] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9125.838906] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9133.040390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9139.122085] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9145.603928] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9291.244579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9291.246211] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9291.257287] Lustre: Skipped 2 previous similar messages [ 9291.263580] LustreError: Skipped 1 previous similar message [ 9291.267088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9291.278700] Lustre: Skipped 10 previous similar messages [ 9305.056120] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9305.193266] Lustre: server umount lustre-OST0000 complete [ 9314.447380] Lustre: server umount lustre-MDT0000 complete [ 9316.834138] LustreError: 245014:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9316.858618] LustreError: 245014:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9318.208229] LustreError: 244993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603010 with bad export cookie 6524559626352124586 [ 9318.210813] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9318.219224] LustreError: 244993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9318.515922] Lustre: server umount lustre-MDT0001 complete [ 9413.927503] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 9416.893189] Key type lgssc unregistered [ 9417.219833] LNet: 248619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9417.231199] LNetError: 248619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9417.251936] LNet: Removed LNI 192.168.203.116@tcp [ 9418.117165] Key type .llcrypt unregistered [ 9418.122125] Key type ._llcrypt unregistered [ 9441.811807] Key type ._llcrypt registered [ 9441.814577] Key type .llcrypt registered [ 9441.937102] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 9458.165262] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9459.172751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9459.220052] alg: No test for adler32 (adler32-zlib) [ 9460.333753] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 9460.645604] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 9462.423232] Key type lgssc registered [ 9463.514784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9509.782553] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9520.938558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9520.952029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9522.101853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9522.161360] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9522.234918] Lustre: lustre-MDT0000: new disk, initializing [ 9522.307163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9522.332250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9526.818169] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9537.523925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9537.657533] Lustre: 252686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9537.700387] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9537.704480] Lustre: Skipped 1 previous similar message [ 9537.761798] Lustre: lustre-MDT0001: new disk, initializing [ 9537.838399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9537.873430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9537.883170] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9542.567789] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9547.894769] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9554.518238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9554.867848] Lustre: lustre-OST0000: new disk, initializing [ 9554.873557] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9554.881658] Lustre: 254315:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9554.990502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9555.346678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9555.368642] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9555.534860] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9562.251874] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9573.716144] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9573.942952] Lustre: lustre-OST0001: new disk, initializing [ 9573.949290] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9573.962473] Lustre: 255198:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9574.118925] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9580.932756] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9582.688228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9582.713744] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9582.797701] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9592.709949] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9603.640384] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9614.824551] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9614.839630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9614.858845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9616.488846] Lustre: server umount lustre-MDT0000 complete [ 9618.925595] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9618.943699] Lustre: Skipped 2 previous similar messages [ 9619.836890] LustreError: 252678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603312 with bad export cookie 5596711180895566385 [ 9619.847398] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9619.853767] LustreError: 252678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9620.299219] Lustre: server umount lustre-MDT0001 complete [ 9634.517796] Lustre: server umount lustre-OST0000 complete [ 9648.919262] Lustre: server umount lustre-OST0001 complete [ 9662.414790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9672.352415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9680.508142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9689.208329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9708.124514] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9717.685170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9717.706788] Lustre: lustre-MDT0000: reset Object Index mappings [ 9717.996064] LustreError: 260209:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9718.007378] LustreError: 260209:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9718.059105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9722.136631] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9723.368037] LustreError: 260210:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9723.388163] LustreError: 260210:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9727.459264] LustreError: 260209:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9730.194349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9730.223335] Lustre: lustre-MDT0001: reset Object Index mappings [ 9730.605225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9734.728605] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9737.392606] Lustre: 261351:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9743.416599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9751.556325] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9756.079667] LustreError: 261709:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9761.252064] LustreError: 261707:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9761.764928] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9762.004621] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9762.009067] Lustre: Skipped 1 previous similar message [ 9763.077211] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9763.077658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9768.370799] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9776.186247] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9779.795832] Lustre: 263226:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9786.849439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9786.866104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9786.923803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9788.898154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9788.923974] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9788.953879] Lustre: Skipped 2 previous similar messages [ 9792.844256] Lustre: server umount lustre-MDT0000 complete [ 9794.030326] LustreError: 260210:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9794.046640] LustreError: 260210:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9796.710769] LustreError: 263228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603489 with bad export cookie 5596711180895573175 [ 9796.721336] LustreError: 263228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9796.722884] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9797.005067] Lustre: server umount lustre-MDT0001 complete [ 9810.550423] Lustre: server umount lustre-OST0000 complete [ 9823.766971] Lustre: server umount lustre-OST0001 complete [ 9846.826216] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 20:05:38 (1789603538) [ 9864.342262] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 9875.630407] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9916.951110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9917.193663] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9917.225112] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9917.321520] Lustre: lustre-MDT0000: new disk, initializing [ 9917.416415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9917.435773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9921.879535] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9934.078955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9934.307774] Lustre: 267481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9934.479542] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9934.494555] Lustre: Skipped 1 previous similar message [ 9934.688525] Lustre: lustre-MDT0001: new disk, initializing [ 9934.902831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9934.921297] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9940.148821] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9952.475780] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9959.394352] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9967.432309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9967.785951] Lustre: lustre-OST0000: new disk, initializing [ 9967.792761] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9967.804710] Lustre: 269454:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9969.092068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9969.097038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9969.278171] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9974.702732] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9983.960258] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9989.011827] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9989.241505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9993.921274] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9994.173507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9999.849153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9999.856563] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9999.903286] LustreError: Skipped 1 previous similar message [ 9999.903686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9999.912736] Lustre: Skipped 4 previous similar messages [10002.538843] Lustre: server umount lustre-OST0000 complete [10010.105587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10010.108408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10010.137653] Lustre: Skipped 2 previous similar messages [10020.320212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10020.330490] Lustre: Skipped 4 previous similar messages [10021.343167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10021.704989] Lustre: server umount lustre-MDT0000 complete [10025.250970] LustreError: 269453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603717 with bad export cookie 5596711180895575268 [10025.262780] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10025.276141] LustreError: 269453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10025.646476] Lustre: server umount lustre-MDT0001 complete [10045.246971] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [10056.495313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10057.010651] LustreError: 272391:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10057.078240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10057.081723] Lustre: Skipped 2 previous similar messages [10061.776956] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10070.871491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10076.707299] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10079.962730] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10086.429088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10093.780821] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10103.055987] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10103.255493] Lustre: lustre-OST0001: new disk, initializing [10103.259591] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10103.267182] Lustre: 274698:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10105.240149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10105.250618] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10105.369856] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10110.720793] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10121.202594] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10124.795551] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10130.363586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10130.430363] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10143.200921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10143.205990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10143.227408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10147.898275] Lustre: server umount lustre-MDT0000 complete [10151.416553] LustreError: 274587:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10151.459703] LustreError: 274587:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10152.205889] LustreError: 274697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603844 with bad export cookie 5596711180895576150 [10152.209802] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10152.223298] LustreError: 274697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10152.678671] Lustre: server umount lustre-MDT0001 complete [10167.258083] Lustre: server umount lustre-OST0000 complete [10181.482294] Lustre: server umount lustre-OST0001 complete [10199.646588] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [10213.239455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10213.986568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10213.992268] Lustre: Skipped 3 previous similar messages [10218.976126] LustreError: 278043:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10218.996373] LustreError: 278043:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10219.564051] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10229.423870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10234.817130] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10238.086533] Lustre: 279186:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10245.272979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10252.161855] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10259.004412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10261.214051] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10266.637470] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10267.837864] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10275.776833] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10279.622158] Lustre: 281064:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10285.153651] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10285.208778] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10297.306087] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 20:13:08 (1789603988) [10312.160691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10312.165788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10312.193105] Lustre: Skipped 3 previous similar messages [10312.197747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10312.203186] Lustre: Skipped 4 previous similar messages [10318.306752] Lustre: server umount lustre-MDT0000 complete [10322.268365] LustreError: 278023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789604014 with bad export cookie 5596711180895582793 [10322.271551] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10322.284885] LustreError: 278023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10328.773594] Lustre: server umount lustre-MDT0001 complete [10338.747089] Lustre: server umount lustre-OST0000 complete [10342.151102] Lustre: server umount lustre-OST0001 complete [10347.719444] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [10356.541270] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [10406.347958] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [10417.667246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10417.905512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10417.951502] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10418.037515] Lustre: lustre-MDT0000: new disk, initializing [10418.228732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10423.609143] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10436.076337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10436.267354] Lustre: 285597:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10436.697359] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10442.511689] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10455.248725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10455.568740] Lustre: 287227:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10456.961786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10461.666595] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10472.582154] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10472.734621] Lustre: 288100:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10472.823284] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10472.826112] Lustre: Skipped 6 previous similar messages [10473.952825] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10473.958790] Lustre: Skipped 1 previous similar message [10474.064575] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10480.105774] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10489.508439] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10492.935726] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10492.942778] Lustre: Skipped 1 previous similar message [10509.722969] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 20:16:40 (1789604200) [10513.379229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10513.391672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10513.407411] Lustre: Skipped 5 previous similar messages [10513.416397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10513.424797] Lustre: Skipped 11 previous similar messages [10518.145811] Lustre: server umount lustre-MDT0000 complete [10520.037151] LustreError: 285608:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10520.074040] LustreError: 285608:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [10521.782450] LustreError: 285590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789604214 with bad export cookie 5596711180895590493 [10521.786138] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10521.795800] LustreError: 285590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10522.291808] Lustre: server umount lustre-MDT0001 complete [10536.252984] Lustre: server umount lustre-OST0000 complete [10550.500480] Lustre: server umount lustre-OST0001 complete [10558.342164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10563.576836] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10571.310874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10576.146499] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10582.813930] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10587.984894] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10593.774402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10599.820492] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10606.184831] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10621.201122] loop1: detected capacity change from 0 to 204800000 [10621.359367] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10621.372784] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10621.383661] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10625.925889] loop1: detected capacity change from 0 to 204800000 [10631.131872] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10631.249438] LustreError: 290338:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs21949: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10631.258951] LustreError: 290338:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs21949 [10631.263775] LustreError: 290338:0:(mgs_handler.c:517:mgs_target_reg()) fs21949-MDT0001: Error checking target index: rc = -13 [10631.269973] LustreError: fs21949-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10631.277331] LustreError: 294068:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10631.286888] LustreError: 294068:0:(tgt_mount.c:2128:server_put_super()) no obd fs21949-MDT0001 [10631.291336] LustreError: 294068:0:(tgt_mount.c:116:server_deregister_mount()) fs21949-MDT0001 not registered [10631.323868] Lustre: server umount fs21949-MDT0001 complete [10631.327803] LustreError: 294068:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10639.023473] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10639.169412] Lustre: Setting parameter fs21949-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs21949-MDT0001 [10639.233398] Lustre: srv-fs21949-MDT0001: No data found on store. Initialize space. [10639.236951] Lustre: Skipped 4 previous similar messages [10639.314684] Lustre: fs21949-MDT0001: new disk, initializing [10639.318178] Lustre: Skipped 3 previous similar messages [10639.419587] Lustre: srv-fs21949-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10639.429810] Lustre: 294535:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21949-MDT0001-mdtlov: can't create new llog: rc = -115 [10639.443662] LustreError: 294535:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21949-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10643.702548] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10649.641286] Lustre: server umount fs21949-MDT0001 complete [10662.441322] loop1: detected capacity change from 0 to 204800000 [10667.668983] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10667.926368] Lustre: 296106:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21949-MDT0001-mdtlov: can't create new llog: rc = -115 [10667.944456] LustreError: 296106:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21949-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10672.000862] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10676.607719] Lustre: server umount fs21949-MDT0001 complete [10680.309377] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10685.411936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10685.417502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10685.430722] Lustre: Skipped 5 previous similar messages [10685.445812] Lustre: Skipped 5 previous similar messages [10691.645654] Lustre: server umount lustre-MDT0000 complete [10699.314136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10699.447827] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10704.037619] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10704.874954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10710.650173] loop1: detected capacity change from 0 to 204800000 [10716.152522] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10716.593103] Lustre: 298549:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21949-MDT0001-mdtlov: can't create new llog: rc = -115 [10716.604886] LustreError: 298549:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21949-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10720.716482] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10725.736325] Lustre: server umount fs21949-MDT0001 complete [10745.828186] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10751.200883] Lustre: server umount lustre-OST0000 complete [10761.990946] Lustre: server umount lustre-MDT0000 complete [10765.786316] LustreError: 290332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789604458 with bad export cookie 5596711180895593174 [10765.792720] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10765.797740] LustreError: 290332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10766.104768] Lustre: server umount lustre-MDT0001 complete [10777.612767] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [10780.848237] Key type lgssc unregistered [10781.270831] LNet: 300891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10781.281348] LNetError: 300891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10781.297456] LNet: Removed LNI 192.168.203.116@tcp [10782.371274] Key type .llcrypt unregistered [10782.375641] Key type ._llcrypt unregistered [10804.098496] Key type ._llcrypt registered [10804.102521] Key type .llcrypt registered [10804.265279] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [10805.273983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10805.333604] alg: No test for adler32 (adler32-zlib) [10806.461337] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [10806.765648] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [10808.504456] Key type lgssc registered [10809.553666] Lustre: Echo OBD driver; http://www.lustre.org/ [10817.173021] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10817.213194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10818.832823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10823.134935] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10829.287320] LustreError: 302162:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10829.301760] LustreError: 302162:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10831.286758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10831.619302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10835.577575] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10842.123892] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10847.416512] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10854.226161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10854.542081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10856.626042] LustreError: 303989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10860.674974] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10862.079464] LustreError: 303992:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10864.179547] LustreError: 303989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10864.217697] LustreError: 303989:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10868.031241] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10869.221077] LustreError: 303991:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10869.245518] LustreError: 303991:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10879.465656] LustreError: 303989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10879.488524] LustreError: 303989:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10884.028488] loop1: detected capacity change from 0 to 204800000 [10884.234423] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10884.251800] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10884.263025] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10889.537721] loop1: detected capacity change from 0 to 204800000 [10894.519631] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10894.678854] Lustre: fs29211-OST0000: new disk, initializing [10894.682165] Lustre: srv-fs29211-OST0000: No data found on store. Initialize space. [10894.690256] Lustre: 305894:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10894.738386] Lustre: fs29211-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10899.948543] LustreError: 303989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10899.982939] LustreError: 303989:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [10900.588348] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10908.383315] Lustre: server umount fs29211-OST0000 complete [10920.051180] loop1: detected capacity change from 0 to 204800000 [10925.437842] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10925.750510] Lustre: fs29211-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10932.010679] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10935.790275] LustreError: 303989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10935.811813] LustreError: 303989:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 20 previous similar messages [10940.113971] loop2: detected capacity change from 0 to 204800000 [10940.406539] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10940.415729] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10940.424425] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10940.921794] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10945.929472] loop2: detected capacity change from 0 to 204800000 [10946.022787] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10951.105674] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10951.213343] LustreError: fs29211-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10951.225048] LustreError: fs29211-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10951.235956] LustreError: 308934:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10951.241860] LustreError: 308934:0:(tgt_mount.c:2128:server_put_super()) no obd fs29211-OST0001 [10951.246294] LustreError: 308934:0:(tgt_mount.c:116:server_deregister_mount()) fs29211-OST0001 not registered [10951.276872] Lustre: server umount fs29211-OST0001 complete [10951.285668] LustreError: 308934:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10961.147650] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10961.377545] Lustre: fs29211-OST0001: new disk, initializing [10961.381415] Lustre: srv-fs29211-OST0001: No data found on store. Initialize space. [10961.395600] Lustre: 309485:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10961.495909] Lustre: fs29211-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10967.311994] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10975.062177] Lustre: server umount fs29211-OST0000 complete [10984.456276] Lustre: server umount fs29211-OST0001 complete [10996.938846] loop1: detected capacity change from 0 to 204800000 [11002.348163] LustreError: 303992:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11002.364403] LustreError: 303992:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [11004.006984] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11004.267700] Lustre: fs29211-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11010.667496] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11017.603746] loop2: detected capacity change from 0 to 204800000 [11023.649061] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11030.379955] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11041.503338] Lustre: server umount fs29211-OST0000 complete [11051.792840] Lustre: server umount fs29211-OST0001 complete [11066.848531] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11066.857124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11066.874387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11068.900414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11068.908107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11068.934063] Lustre: Skipped 1 previous similar message [11074.017862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11074.032153] Lustre: Skipped 1 previous similar message [11078.113569] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11078.336909] Lustre: server umount lustre-OST0000 complete [11082.740118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11082.753825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11082.779516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11088.339279] Lustre: server umount lustre-MDT0000 complete [11092.303673] LustreError: 302144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789604784 with bad export cookie 8653168894957696022 [11092.312890] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11092.319081] LustreError: 302144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11092.743647] Lustre: server umount lustre-MDT0001 complete [11103.678412] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [11106.725889] Key type lgssc unregistered [11107.031948] LNet: 315411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11107.043343] LNetError: 315411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11107.061808] LNet: Removed LNI 192.168.203.116@tcp [11108.097176] Key type .llcrypt unregistered [11108.104506] Key type ._llcrypt unregistered [11142.060368] Key type ._llcrypt registered [11142.062451] Key type .llcrypt registered [11142.194169] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [11158.659617] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [11159.823330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11160.000485] alg: No test for adler32 (adler32-zlib) [11161.134694] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [11161.418956] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [11163.103911] Key type lgssc registered [11164.024632] Lustre: Echo OBD driver; http://www.lustre.org/ [11200.846847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11200.880933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11202.236292] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11202.288695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11202.451628] Lustre: lustre-MDT0000: new disk, initializing [11202.601435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11202.627791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11207.121782] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11218.951452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11219.085316] Lustre: 319270:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11219.154885] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11219.161880] Lustre: Skipped 1 previous similar message [11219.290516] Lustre: lustre-MDT0001: new disk, initializing [11219.439445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11219.528416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11219.558572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11224.902313] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11235.786464] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11242.432520] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11249.953137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11250.257670] Lustre: lustre-OST0000: new disk, initializing [11250.264437] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11250.273992] Lustre: 321238:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11250.403381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11256.752621] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11256.946401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11256.959352] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11257.149884] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11266.686323] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11268.831558] Lustre: Mounted lustre-client - version 2.17.58_39_gf14a6ef [11269.959518] Lustre: Unmounted lustre-client [11279.335525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11279.347534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11279.348733] Lustre: Skipped 1 previous similar message [11279.358607] Lustre: Skipped 1 previous similar message [11284.457600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11284.464868] Lustre: Skipped 1 previous similar message [11285.310141] Lustre: server umount lustre-OST0000 complete [11289.571915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11289.590317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11289.593150] Lustre: Skipped 1 previous similar message [11289.604405] Lustre: Skipped 1 previous similar message [11294.689858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11294.696353] Lustre: Skipped 1 previous similar message [11296.850935] Lustre: server umount lustre-MDT0000 complete [11300.178996] LustreError: 319260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789604992 with bad export cookie 16408738925243897852 [11300.182689] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11300.186820] LustreError: 319260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11300.555416] Lustre: server umount lustre-MDT0001 complete [11311.199374] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [11313.884154] Key type lgssc unregistered [11330.400337] LNet: 1 local NIs in recovery (showing 1): 192.168.203.116@tcp [11330.591639] LNet: 323399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11330.614468] LNetError: 323399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11400.294403] LNet: Removed LNI 192.168.203.116@tcp [11401.046430] Key type .llcrypt unregistered [11401.050951] Key type ._llcrypt unregistered [11418.884682] Key type ._llcrypt registered [11418.886507] Key type .llcrypt registered [11419.044345] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [11434.025170] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [11435.089077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11435.121103] alg: No test for adler32 (adler32-zlib) [11436.100858] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [11436.349789] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [11438.047306] Key type lgssc registered [11439.013643] Lustre: Echo OBD driver; http://www.lustre.org/ [11465.898751] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 20:32:37 (1789605157) [11471.597077] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11471.630451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11472.815552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11472.844567] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11472.935311] Lustre: lustre-MDT0000: new disk, initializing [11473.023427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11473.042286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11477.530782] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11487.314690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11487.440592] Lustre: 326618:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11487.480419] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11487.483565] Lustre: Skipped 1 previous similar message [11487.534665] Lustre: lustre-MDT0001: new disk, initializing [11487.607511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11487.634934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11487.640880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11491.678077] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11499.933648] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11505.677236] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11511.880468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11512.214652] Lustre: lustre-OST0000: new disk, initializing [11512.222707] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11512.234869] Lustre: 328584:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11512.338961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11517.523967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11517.544239] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11517.690141] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11519.441901] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11529.856733] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11548.644449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11548.645177] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11548.662771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11548.673130] Lustre: Skipped 1 previous similar message [11548.704758] LustreError: Skipped 1 previous similar message [11551.387354] Lustre: server umount lustre-OST0000 complete [11558.880565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11558.884820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11558.898580] Lustre: Skipped 1 previous similar message [11558.934104] Lustre: Skipped 2 previous similar messages [11561.246897] Lustre: server umount lustre-MDT0000 complete [11565.064087] LustreError: 326610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605257 with bad export cookie 10634661995499876148 [11565.086079] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11565.407440] Lustre: server umount lustre-MDT0001 complete [11579.158371] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [11583.189140] Key type lgssc unregistered [11583.567300] LNet: 330493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11583.588382] LNetError: 330493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11583.618533] LNet: Removed LNI 192.168.203.116@tcp [11584.883194] Key type .llcrypt unregistered [11584.893936] Key type ._llcrypt unregistered [11608.973885] Key type ._llcrypt registered [11608.980976] Key type .llcrypt registered [11609.103207] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [11610.333678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11610.350726] alg: No test for adler32 (adler32-zlib) [11611.332547] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [11611.652868] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [11613.335422] Key type lgssc registered [11614.251968] Lustre: Echo OBD driver; http://www.lustre.org/ [11621.471348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11621.503257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11622.886299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11626.826318] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11633.128460] LustreError: 331769:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11633.147811] LustreError: 331769:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11634.625545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11635.058483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11639.322386] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11646.155044] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11652.337302] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11659.044189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11659.509619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11666.100590] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11672.876892] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11682.592506] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11684.236731] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11688.418308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11688.420665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11688.429263] Lustre: Skipped 2 previous similar messages [11688.440628] Lustre: Skipped 2 previous similar messages [11690.905193] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.16@tcp (stopping) [11693.539634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11693.545932] Lustre: Skipped 2 previous similar messages [11696.002520] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.16@tcp (stopping) [11701.123788] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.16@tcp (stopping) [11701.139427] Lustre: Skipped 3 previous similar messages [11702.239063] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11702.610507] Lustre: server umount lustre-MDT0000 complete [11703.787765] LustreError: 331769:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11706.244857] LustreError: 331764:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11706.288275] LustreError: 331764:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11707.010426] LustreError: 331751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605399 with bad export cookie 9930834425102822267 [11707.030051] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11708.896792] LustreError: 331765:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11708.912264] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11711.381241] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.16@tcp (stopping) [11711.387430] Lustre: Skipped 1 previous similar message [11712.931798] LustreError: 331765:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11712.946274] LustreError: 331765:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11713.452360] Lustre: server umount lustre-MDT0001 complete [11718.068956] Lustre: server umount lustre-OST0000 complete [11728.433852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11733.105775] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11735.931907] Lustre: 335951:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.116@tcp [11738.036493] Lustre: server umount MGS complete [11745.754535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11746.145789] LustreError: 336523:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11746.216927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11749.909758] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11756.871882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11757.201875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11760.770970] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11768.177805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11768.419547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11774.407676] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11781.539461] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11782.996694] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11788.334203] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11789.751370] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11795.942722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11795.955061] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11795.974825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11795.981842] Lustre: Skipped 1 previous similar message [11797.147163] Lustre: server umount lustre-OST0000 complete [11802.595454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11802.607661] Lustre: Skipped 1 previous similar message [11813.856528] Lustre: 331080:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789605490/real 1789605490] req@00000000582e5496 x1876537171307776/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789605506 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11813.876804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11815.391249] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11815.632874] Lustre: server umount lustre-MDT0000 complete [11818.979985] LustreError: 336524:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11819.001521] LustreError: 336524:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11819.022725] LustreError: 336505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605511 with bad export cookie 9930834425102823107 [11819.023526] Lustre: 331080:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789605495/real 1789605495] req@000000001072735a x1876537171310080/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789605511 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11819.034872] LustreError: 336505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11819.037663] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11819.379549] Lustre: server umount lustre-MDT0001 complete [11830.172130] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [11832.844875] Key type lgssc unregistered [11833.153974] LNet: 339956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11833.177663] LNetError: 339956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11833.193037] LNet: Removed LNI 192.168.203.116@tcp [11833.951877] Key type .llcrypt unregistered [11833.954128] Key type ._llcrypt unregistered [11851.195108] Key type ._llcrypt registered [11851.198515] Key type .llcrypt registered [11851.312321] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [11866.319234] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [11867.410278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11867.450237] alg: No test for adler32 (adler32-zlib) [11868.498348] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [11868.806365] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [11870.527914] Key type lgssc registered [11871.556798] Lustre: Echo OBD driver; http://www.lustre.org/ [11905.827139] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11905.910704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11907.254888] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11907.310880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11907.413582] Lustre: lustre-MDT0000: new disk, initializing [11907.488805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11907.516929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11911.635273] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11921.201973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11921.360502] Lustre: 343113:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11921.418407] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11921.423371] Lustre: Skipped 1 previous similar message [11921.520858] Lustre: lustre-MDT0001: new disk, initializing [11921.634614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11921.672026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11921.683188] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11925.367944] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11933.640588] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11938.579681] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11944.468502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11944.710628] Lustre: lustre-OST0000: new disk, initializing [11944.713808] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11944.724556] Lustre: 345083:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11944.806118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11950.754345] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11951.665963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11951.685408] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11951.792757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11960.142247] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11964.631838] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11964.898642] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11970.473652] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11970.646337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11977.704823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11977.712686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11977.715700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11977.719535] LustreError: Skipped 1 previous similar message [11977.753751] Lustre: Skipped 1 previous similar message [11979.056933] Lustre: server umount lustre-OST0000 complete [11982.820914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11982.825295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11982.839378] Lustre: Skipped 1 previous similar message [11982.854282] Lustre: Skipped 2 previous similar messages [11987.937938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11987.950906] Lustre: Skipped 1 previous similar message [11988.924362] Lustre: server umount lustre-MDT0000 complete [11993.360440] LustreError: 343105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605686 with bad export cookie 4317297208164870031 [11993.361130] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11993.371127] LustreError: 343105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11993.686866] Lustre: server umount lustre-MDT0001 complete [12004.691295] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 20:41:35 (1789605695) [12021.465412] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [12030.574781] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [12063.583116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12063.797745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12063.840816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12063.936405] Lustre: lustre-MDT0000: new disk, initializing [12064.011375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12064.031754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12069.103434] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12078.941081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12079.046825] Lustre: 349840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12079.084858] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12079.088306] Lustre: Skipped 1 previous similar message [12079.145306] Lustre: lustre-MDT0001: new disk, initializing [12079.243078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12079.288688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12079.303701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12083.786849] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12094.154898] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12099.430654] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12105.719116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12106.063474] Lustre: lustre-OST0000: new disk, initializing [12106.070280] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12106.081475] Lustre: 351808:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12106.169966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12112.162385] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12115.006232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12115.022758] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12115.097898] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12120.734222] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12151.271362] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12151.272567] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12151.281602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12151.300283] Lustre: Skipped 1 previous similar message [12155.594521] Lustre: server umount lustre-OST0000 complete [12160.992968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12161.010730] LustreError: Skipped 1 previous similar message [12161.029445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12161.057210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12161.067028] Lustre: Skipped 1 previous similar message [12165.677337] Lustre: server umount lustre-MDT0000 complete [12166.638740] LustreError: 349854:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12166.676739] LustreError: 349854:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12169.406923] LustreError: 350646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605862 with bad export cookie 4317297208164870913 [12169.412711] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12169.419281] LustreError: 350646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12169.830487] Lustre: server umount lustre-MDT0001 complete [12181.570307] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12184.767957] Key type lgssc unregistered [12185.032609] LNet: 353960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12185.044770] LNetError: 353960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12185.061082] LNet: Removed LNI 192.168.203.116@tcp [12185.988813] Key type .llcrypt unregistered [12185.991966] Key type ._llcrypt unregistered [12272.911121] Key type ._llcrypt registered [12272.919223] Key type .llcrypt registered [12273.028411] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12273.238752] Key type .llcrypt unregistered [12273.243784] Key type ._llcrypt unregistered [12347.395486] Key type ._llcrypt registered [12347.397307] Key type .llcrypt registered [12347.501589] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12347.834301] Key type .llcrypt unregistered [12347.840188] Key type ._llcrypt unregistered [12376.316432] Key type ._llcrypt registered [12376.319772] Key type .llcrypt registered [12376.420409] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12376.668392] Key type .llcrypt unregistered [12376.670943] Key type ._llcrypt unregistered [12405.013803] Key type ._llcrypt registered [12405.015894] Key type .llcrypt registered [12405.157977] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12405.484503] Key type .llcrypt unregistered [12405.487959] Key type ._llcrypt unregistered [12479.261950] Key type ._llcrypt registered [12479.264695] Key type .llcrypt registered [12479.418778] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12479.740111] Key type .llcrypt unregistered [12479.744688] Key type ._llcrypt unregistered [12509.664809] Key type ._llcrypt registered [12509.672579] Key type .llcrypt registered [12509.779567] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12510.064546] Key type .llcrypt unregistered [12510.066270] Key type ._llcrypt unregistered [12537.752979] Key type ._llcrypt registered [12537.756992] Key type .llcrypt registered [12537.890074] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12538.177902] Key type .llcrypt unregistered [12538.184610] Key type ._llcrypt unregistered [12571.781440] Key type ._llcrypt registered [12571.783556] Key type .llcrypt registered [12571.920354] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12572.187550] Key type .llcrypt unregistered [12572.190659] Key type ._llcrypt unregistered [12604.289362] Key type ._llcrypt registered [12604.299339] Key type .llcrypt registered [12604.410538] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12604.737373] Key type .llcrypt unregistered [12604.744128] Key type ._llcrypt unregistered [12634.679872] Key type ._llcrypt registered [12634.684078] Key type .llcrypt registered [12634.895833] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12635.211458] Key type .llcrypt unregistered [12635.214615] Key type ._llcrypt unregistered [12700.958299] Key type ._llcrypt registered [12700.969031] Key type .llcrypt registered [12701.110957] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12701.489315] Key type .llcrypt unregistered [12701.496182] Key type ._llcrypt unregistered [12731.059499] Key type ._llcrypt registered [12731.060883] Key type .llcrypt registered [12731.176898] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12731.520970] Key type .llcrypt unregistered [12731.523897] Key type ._llcrypt unregistered [12766.807951] Key type ._llcrypt registered [12766.813908] Key type .llcrypt registered [12766.939336] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12767.266473] Key type .llcrypt unregistered [12767.268893] Key type ._llcrypt unregistered [12819.313035] Key type ._llcrypt registered [12819.314888] Key type .llcrypt registered [12819.451842] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [12834.637231] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [12835.920647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12836.178412] alg: No test for adler32 (adler32-zlib) [12837.281584] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [12837.526923] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [12839.215195] Key type lgssc registered [12840.435974] Lustre: Echo OBD driver; http://www.lustre.org/ [12874.627772] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12874.677749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12875.913236] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12875.939209] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12876.009182] Lustre: tst250fs-MDT0000: new disk, initializing [12876.106376] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12876.127082] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12880.729522] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12891.892984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12892.058543] Lustre: 374236:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12892.123627] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12892.128036] Lustre: Skipped 1 previous similar message [12892.242646] Lustre: tst250fs-MDT0001: new disk, initializing [12892.391147] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12892.476416] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12892.498179] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12897.672646] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12903.210269] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12910.532663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12910.926237] Lustre: tst250fs-OST0000: new disk, initializing [12910.940807] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12910.950185] Lustre: 375870:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12911.084764] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12911.367864] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12911.398843] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12911.436778] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [12918.196417] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12929.035308] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12929.363103] Lustre: tst250fs-OST0001: new disk, initializing [12929.372258] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12929.387386] Lustre: 376742:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12929.503479] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12935.462798] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12937.323101] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12937.333543] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12937.420243] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12941.228384] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12943.382772] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [12943.388780] Lustre: Skipped 1 previous similar message [12963.299563] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12963.306900] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12963.331979] Lustre: Skipped 3 previous similar messages [12963.346476] Lustre: Skipped 2 previous similar messages [12967.745932] Lustre: server umount tst250fs-MDT0000 complete [12970.672185] LustreError: 374230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789606663 with bad export cookie 9731165207674865809 [12970.677358] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12970.680918] LustreError: 374230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12970.998356] Lustre: server umount tst250fs-MDT0001 complete [12983.651563] Lustre: server umount tst250fs-OST0000 complete [12998.104597] Lustre: server umount tst250fs-OST0001 complete [13021.339121] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [13034.071299] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [13063.852721] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 20:59:15 (1789606755) [13065.120031] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13079.904934] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [13089.818350] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [13121.114913] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12779 sec ========= 21:00:11 (1789606811) [13122.829511] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:00:14 (1789606814) === [13126.315766] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:00:17 (1789606817) === [13172.934665] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [13176.119463] Key type lgssc unregistered [13176.354896] LNet: 385901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13176.359661] LNetError: 385901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13177.383044] LNet: Removed LNI 192.168.203.116@tcp [13178.637155] Key type .llcrypt unregistered [13178.640635] Key type ._llcrypt unregistered