[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 549125856 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003356] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008032] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009021] pid_max: default: 32768 minimum: 301 [ 0.010174] LSM: Security Framework initializing [ 0.011116] Yama: becoming mindful. [ 0.012070] SELinux: Initializing. [ 0.013118] *** VALIDATE selinux *** [ 0.021538] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026323] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027203] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028144] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030164] *** VALIDATE tmpfs *** [ 0.032170] *** VALIDATE proc *** [ 0.033000] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.035313] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036178] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038039] Spectre V2 : User space: Vulnerable [ 0.039016] Speculative Store Bypass: Vulnerable [ 0.042482] debug: unmapping init [mem 0xffffffffae859000-0xffffffffae860fff] [ 0.044215] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045816] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046037] ... version: 2 [ 0.047020] ... bit width: 48 [ 0.048020] ... generic registers: 4 [ 0.049020] ... value mask: 0000ffffffffffff [ 0.050024] ... max period: 00007fffffffffff [ 0.051023] ... fixed-purpose events: 3 [ 0.052019] ... event mask: 000000070000000f [ 0.053376] rcu: Hierarchical SRCU implementation. [ 0.055872] smp: Bringing up secondary CPUs ... [ 0.056723] x86: Booting SMP configuration: [ 0.057034] .... node #0, CPUs: #1 #2 #3 [ 0.061219] smp: Brought up 1 node, 4 CPUs [ 0.063022] smpboot: Max logical packages: 1 [ 0.064020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.226035] node 0 deferred pages initialised in 160ms [ 0.233174] devtmpfs: initialized [ 0.234311] x86/mm: Memory block size: 128MB [ 0.239192] gcov: version magic: 0x41383552 [ 0.242577] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.249108] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.256462] pinctrl core: initialized pinctrl subsystem [ 0.259189] [ 0.261011] ************************************************************* [ 0.264014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.267014] ** ** [ 0.270017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.273015] ** ** [ 0.276015] ** This means that this kernel is built to expose internal ** [ 0.280015] ** IOMMU data structures, which may compromise security on ** [ 0.282017] ** your system. ** [ 0.285013] ** ** [ 0.288012] ** If you see this message and you are not debugging the ** [ 0.290016] ** kernel, report this immediately to your vendor! ** [ 0.293015] ** ** [ 0.295015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.298016] ************************************************************* [ 0.300722] NET: Registered protocol family 16 [ 0.302364] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.305076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.308054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.312047] cpuidle: using governor menu [ 0.314023] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.316696] PCI: Using configuration type 1 for base access [ 0.319154] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.331271] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.332000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.335482] cryptd: max_cpu_qlen set to 1000 [ 0.340094] ACPI: Added _OSI(Module Device) [ 0.341000] ACPI: Added _OSI(Processor Device) [ 0.341019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.343131] ACPI: Added _OSI(Processor Aggregator Device) [ 0.350149] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.360628] ACPI: Interpreter enabled [ 0.362092] ACPI: PM: (supports S0 S3 S4 S5) [ 0.364018] ACPI: Using IOAPIC for interrupt routing [ 0.367199] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.372428] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.385352] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.388062] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.391026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.396106] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.403020] acpiphp: Slot [2] registered [ 0.404366] acpiphp: Slot [5] registered [ 0.407214] acpiphp: Slot [6] registered [ 0.409185] acpiphp: Slot [7] registered [ 0.411186] acpiphp: Slot [8] registered [ 0.413196] acpiphp: Slot [9] registered [ 0.415209] acpiphp: Slot [10] registered [ 0.416322] acpiphp: Slot [3] registered [ 0.418392] acpiphp: Slot [4] registered [ 0.420366] acpiphp: Slot [11] registered [ 0.422266] acpiphp: Slot [12] registered [ 0.423146] acpiphp: Slot [13] registered [ 0.424185] acpiphp: Slot [14] registered [ 0.426155] acpiphp: Slot [15] registered [ 0.427245] acpiphp: Slot [16] registered [ 0.429135] acpiphp: Slot [17] registered [ 0.431141] acpiphp: Slot [18] registered [ 0.433158] acpiphp: Slot [19] registered [ 0.435226] acpiphp: Slot [20] registered [ 0.436246] acpiphp: Slot [21] registered [ 0.438176] acpiphp: Slot [22] registered [ 0.441128] acpiphp: Slot [23] registered [ 0.443198] acpiphp: Slot [24] registered [ 0.445179] acpiphp: Slot [25] registered [ 0.446000] acpiphp: Slot [26] registered [ 0.447302] acpiphp: Slot [27] registered [ 0.449322] acpiphp: Slot [28] registered [ 0.451138] acpiphp: Slot [29] registered [ 0.453341] acpiphp: Slot [30] registered [ 0.455158] acpiphp: Slot [31] registered [ 0.457124] PCI host bridge to bus 0000:00 [ 0.459031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.462038] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.465143] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.469042] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.473097] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.476042] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.479203] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.483343] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.487922] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.498987] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.504549] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.508021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.510020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.513024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.516459] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.519104] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.522071] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.527144] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.534021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.548022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.554022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.560284] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.567018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.574018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.588035] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.598708] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.607000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.614023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.646025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.661380] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.668020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.674022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.692019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.703139] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.711018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.718020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.749020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.760000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.769060] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.774022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.793021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.806353] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.815019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.822019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.845020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.861166] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.863431] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.866433] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.868436] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.871280] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.878325] iommu: Default domain type: Passthrough [ 0.880490] SCSI subsystem initialized [ 0.882232] ACPI: bus type USB registered [ 0.883182] usbcore: registered new interface driver usbfs [ 0.886106] usbcore: registered new interface driver hub [ 0.888173] usbcore: registered new device driver usb [ 0.890233] pps_core: LinuxPPS API ver. 1 registered [ 0.892013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.896071] PTP clock support registered [ 0.898140] EDAC MC: Ver: 3.0.0 [ 0.900128] PCI: Using ACPI for IRQ routing [ 0.903439] NetLabel: Initializing [ 0.905012] NetLabel: domain hash size = 128 [ 0.907013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.909093] NetLabel: unlabeled traffic allowed by default [ 0.911144] vgaarb: loaded [ 0.914144] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.916012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.923496] clocksource: Switched to clocksource kvm-clock [ 1.067217] VFS: Disk quotas dquot_6.6.0 [ 1.068938] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.072365] *** VALIDATE ramfs *** [ 1.073912] *** VALIDATE hugetlbfs *** [ 1.077975] pnp: PnP ACPI init [ 1.080559] pnp: PnP ACPI: found 6 devices [ 1.095993] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.099871] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.102562] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.105043] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.107570] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.109900] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.113429] NET: Registered protocol family 2 [ 1.116199] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.121339] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.125199] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.132747] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.137297] TCP: Hash tables configured (established 65536 bind 65536) [ 1.141093] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.146896] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.149681] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.153109] NET: Registered protocol family 1 [ 1.155907] RPC: Registered named UNIX socket transport module. [ 1.158408] RPC: Registered udp transport module. [ 1.160184] RPC: Registered tcp transport module. [ 1.161988] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.164565] NET: Registered protocol family 44 [ 1.166198] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.168472] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.172652] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.176217] PCI: CLS 0 bytes, default 64 [ 1.180463] Unpacking initramfs... [ 2.601239] debug: unmapping init [mem 0xffff98bf7cc54000-0xffff98bf7ffbffff] [ 2.608104] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.610453] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.613683] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.133739] Initialise system trusted keyrings [ 3.135441] Key type blacklist registered [ 3.139063] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.156366] zbud: loaded [ 3.161292] *** VALIDATE nfs *** [ 3.162544] *** VALIDATE nfs4 *** [ 3.164980] pstore: using deflate compression [ 3.169775] Platform Keyring initialized [ 3.283350] NET: Registered protocol family 38 [ 3.284989] Key type asymmetric registered [ 3.286269] Asymmetric key parser 'x509' registered [ 3.287761] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.290755] io scheduler mq-deadline registered [ 3.293396] io scheduler kyber registered [ 3.295627] io scheduler bfq registered [ 3.298203] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.302246] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.305337] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.308664] ACPI: Power Button [PWRF] [ 3.314764] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.323202] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.342606] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.353376] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.383160] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.412875] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.442523] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.448456] Non-volatile memory driver v1.3 [ 3.450850] Linux agpgart interface v0.103 [ 3.487363] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.490384] vda: detected capacity change from 0 to 76677120 [ 3.513752] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.517490] vdb: detected capacity change from 0 to 1073741824 [ 3.533352] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.537166] vdc: detected capacity change from 0 to 2621440000 [ 3.555900] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.559914] vdd: detected capacity change from 0 to 2621440000 [ 3.577918] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.580238] vde: detected capacity change from 0 to 4294967296 [ 3.603716] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.607180] vdf: detected capacity change from 0 to 4294967296 [ 3.615891] libphy: Fixed MDIO Bus: probed [ 3.631201] usbcore: registered new interface driver usbserial_generic [ 3.636168] usbserial: USB Serial support registered for generic [ 3.638898] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.643857] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.647326] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.650104] mousedev: PS/2 mouse device common for all mice [ 3.653520] rtc_cmos 00:05: RTC can wake from S4 [ 3.660524] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.661286] rtc_cmos 00:05: registered as rtc0 [ 3.666234] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.670070] intel_pstate: CPU model not supported [ 3.671261] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.675568] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.675980] hid: raw HID events driver (C) Jiri Kosina [ 3.681162] usbcore: registered new interface driver usbhid [ 3.683648] usbhid: USB HID core driver [ 3.685673] drop_monitor: Initializing network drop monitor service [ 3.688813] Initializing XFRM netlink socket [ 3.691673] NET: Registered protocol family 10 [ 3.695427] Segment Routing with IPv6 [ 3.696838] NET: Registered protocol family 17 [ 3.698853] mpls_gso: MPLS GSO support [ 3.705198] RAS: Correctable Errors collector initialized. [ 3.707313] AVX version of gcm_enc/dec engaged. [ 3.711448] AES CTR mode by8 optimization enabled [ 3.798097] sched_clock: Marking stable (3798066704, 0)->(4841625937, -1043559233) [ 3.802108] registered taskstats version 1 [ 3.804162] Loading compiled-in X.509 certificates [ 3.806665] zswap: loaded using pool lzo/zbud [ 3.835834] Key type big_key registered [ 3.849920] Key type encrypted registered [ 3.852441] ima: No TPM chip found, activating TPM-bypass! [ 3.855164] ima: Allocated hash algorithm: sha1 [ 3.857382] ima: No architecture policies found [ 3.859723] evm: Initialising EVM extended attributes: [ 3.862245] evm: security.selinux [ 3.863584] evm: security.ima [ 3.864756] evm: security.capability [ 3.866474] evm: HMAC attrs: 0x1 [ 3.869074] rtc_cmos 00:05: setting system clock to 2026-09-03 12:02:34 UTC (1788436954) [ 3.875283] debug: unmapping init [mem 0xffffffffaf803000-0xffffffffaf9fffff] [ 3.878374] debug: unmapping init [mem 0xffffffffae582000-0xffffffffae858fff] [ 3.887082] Write protecting the kernel read-only data: 28672k [ 3.890573] debug: unmapping init [mem 0xffffffffacc03000-0xffffffffacdfffff] [ 3.893253] debug: unmapping init [mem 0xffffffffad514000-0xffffffffad5fffff] [ 3.927594] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.938546] systemd[1]: Detected virtualization kvm. [ 3.940629] systemd[1]: Detected architecture x86-64. [ 3.942307] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.966588] systemd[1]: No hostname configured. [ 3.968554] systemd[1]: Set hostname to . [ 3.970779] random: systemd: uninitialized urandom read (16 bytes read) [ 3.973518] systemd[1]: Initializing machine ID from random generator. [ 4.038379] random: ln: uninitialized urandom read (6 bytes read) [ 4.133141] random: systemd: uninitialized urandom read (16 bytes read) [ 4.136740] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.142505] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.147677] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.808669] device-mapper: uevent: version 1.0.3 [ 4.811344] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.585639] virtio_net virtio0 ens2: renamed from eth0 [ 5.600488] random: fast init done Starting dracut initqueue hook... [ 5.858303] scsi host0: ata_piix [ 5.863548] scsi host1: ata_piix [ 5.866173] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.870081] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.103123] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.372584] random: crng init done [ 10.373980] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.901704] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.135120] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.383465] SELinux: Disabled at runtime. [ 12.438173] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.448103] systemd[1]: Detected virtualization kvm. [ 12.449739] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.982565] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.986643] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.991915] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.996637] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.000618] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.012491] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.024401] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Reached target RPC Port Mapper. [ 13.110053] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.439438] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.887275] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.904352] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.033991] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.051805] EDAC sbridge: Ver: 1.1.2 [ 15.904321] Key type dns_resolver registered [ 16.215290] NFS: Registering the id_resolver key type [ 16.217645] Key type id_resolver registered [ 16.219638] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Started Authorization Manager. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg637-server login: [ 47.525692] libcfs: loading out-of-tree module taints kernel. [ 47.590554] Key type ._llcrypt registered [ 47.597246] Key type .llcrypt registered [ 47.713660] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_hostid [ 67.852277] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 70.318698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 70.334530] alg: No test for adler32 (adler32-zlib) [ 72.143517] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 73.236903] LNet: Added LNI 192.168.206.137@tcp [8/256/0/180] [ 74.967865] Key type lgssc registered [ 77.335827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 99.804040] hrtimer: interrupt took 13909691 ns [ 100.831697] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 145.119941] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 160.972064] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 161.049602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 162.356204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 162.397150] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 162.486866] Lustre: lustre-MDT0000: new disk, initializing [ 162.585478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 162.602848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 167.355137] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.705933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 183.792187] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 183.808948] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 183.812163] Lustre: Skipped 1 previous similar message [ 183.872170] Lustre: lustre-MDT0001: new disk, initializing [ 183.938748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 183.981736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 184.002086] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 190.178527] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.376479] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 206.260639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 206.685771] Lustre: lustre-OST0000: new disk, initializing [ 206.690629] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 206.706631] Lustre: 8449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 206.794138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 213.288981] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.118628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 215.126491] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 215.187829] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 226.844342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 226.968528] Lustre: lustre-OST0001: new disk, initializing [ 226.971519] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 226.976301] Lustre: 9523:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 227.036786] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 232.968373] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.556230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 235.574967] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 235.642148] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 243.966831] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 252.307551] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 259.072531] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing check_logdir /tmp/testlogs/ [ 265.006595] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing yml_node [ 269.418772] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 272.346596] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 275.055770] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 276.710396] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:07:06 EDT 2026 [ 294.459766] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 296.129606] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 312.289564] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.291427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 312.312418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 317.409782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 317.429025] Lustre: Skipped 6 previous similar messages [ 318.203938] Lustre: server umount lustre-MDT0000 complete [ 321.829686] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437272 with bad export cookie 16756819319782113388 [ 321.835899] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 321.841612] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 322.528776] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 322.532071] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 322.549589] Lustre: Skipped 4 previous similar messages [ 322.564930] Lustre: Skipped 1 previous similar message [ 326.561225] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 326.565270] Lustre: Skipped 1 previous similar message [ 331.692930] LustreError: 6518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 331.693395] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 331.713874] LustreError: 6518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 331.738807] Lustre: Skipped 1 previous similar message [ 336.351353] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 336.558224] Lustre: server umount lustre-MDT0001 complete [ 340.687178] Lustre: server umount lustre-OST0000 complete [ 344.775869] Lustre: server umount lustre-OST0001 complete [ 351.103431] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_hostid [ 360.683248] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 396.067679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.289750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 396.315889] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 396.392454] Lustre: lustre-MDT0000: new disk, initializing [ 396.453564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 396.467221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 401.635185] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 412.687271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.778895] Lustre: 15075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 412.863256] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 412.866643] Lustre: Skipped 1 previous similar message [ 412.954322] Lustre: lustre-MDT0001: new disk, initializing [ 413.110576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 413.158025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 413.175373] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 417.431532] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 427.243850] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 433.023862] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 439.747205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 440.119913] Lustre: lustre-OST0000: new disk, initializing [ 440.125343] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 440.134201] Lustre: 17043:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 440.269354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 441.611690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 441.627246] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 441.790792] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 446.854250] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 455.502547] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 459.888903] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 460.165303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 465.075963] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 465.322980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 467.438251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.439831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 467.459698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 467.490584] Lustre: Skipped 1 previous similar message [ 467.523441] LustreError: Skipped 1 previous similar message [ 473.634551] Lustre: server umount lustre-OST0000 complete [ 477.665700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 477.672298] Lustre: Skipped 1 previous similar message [ 483.912766] Lustre: server umount lustre-MDT0000 complete [ 487.566376] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437438 with bad export cookie 16756819319782114928 [ 487.567347] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 487.574038] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 487.930529] Lustre: server umount lustre-MDT0001 complete [ 493.334943] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 08:10:42 (1788437442) [ 508.698270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.141891] LustreError: 19916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.202623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 513.409677] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 514.531875] LustreError: 19917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.627284] LustreError: 19916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.487546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 527.191726] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 535.426497] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 541.796383] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 544.006471] Lustre: Failing over lustre-MDT0000 [ 544.227418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 544.241066] LustreError: 19916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.419057] Lustre: server umount lustre-MDT0000 complete [ 552.417238] LustreError: 19911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 552.431337] LustreError: 19911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 562.591185] Lustre: 3640:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788437497/real 1788437497] req@ffff98bec273ea00 x1875312139987456/t0(0) o400->MGC192.168.206.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788437513 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 562.643329] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 564.049851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 572.896033] LustreError: 3637:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff98bec3695c00 x1875312139989760/t0(0) o250->MGC192.168.206.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 572.935950] LustreError: 20859:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.970026] LustreError: 20859:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 578.115559] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 578.537640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 578.552305] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 578.578717] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 587.697776] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 592.609074] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 592.612801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 592.625699] Lustre: Skipped 2 previous similar messages [ 592.639541] Lustre: Skipped 8 previous similar messages [ 597.910267] Lustre: server umount lustre-MDT0000 complete [ 601.849439] LustreError: 19899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437552 with bad export cookie 16756819319782116342 [ 601.862320] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 601.867104] LustreError: 19899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 602.297115] Lustre: server umount lustre-MDT0001 complete [ 611.024618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 611.509824] LustreError: 23253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 611.523969] LustreError: 23253:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 611.587240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 611.593442] Lustre: Skipped 2 previous similar messages [ 615.673352] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 625.348759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 630.945850] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 639.159525] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 645.002455] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 647.233758] Lustre: Failing over lustre-MDT0000 [ 647.778452] Lustre: server umount lustre-MDT0000 complete [ 651.745046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 667.053345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 667.156825] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 667.485848] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 671.966590] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 672.556333] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 672.561367] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 672.584034] Lustre: Skipped 1 previous similar message [ 672.606936] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 681.448940] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 687.072283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 687.074714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 687.085498] Lustre: Skipped 2 previous similar messages [ 687.098375] Lustre: Skipped 1 previous similar message [ 691.512498] Lustre: server umount lustre-MDT0000 complete [ 692.194869] LustreError: 23254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 692.221863] LustreError: 23254:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 694.871847] LustreError: 23234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437645 with bad export cookie 16756819319782117280 [ 694.875491] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 694.892487] LustreError: 23234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 695.208952] Lustre: server umount lustre-MDT0001 complete [ 703.244635] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 08:14:12 (1788437652) [ 719.887512] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_hostid [ 729.660749] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 764.544490] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 774.501422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 774.733182] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 774.772140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 774.840677] Lustre: lustre-MDT0000: new disk, initializing [ 774.909672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 774.919400] Lustre: Skipped 2 previous similar messages [ 774.938291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 778.868602] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 787.674645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 787.807138] Lustre: 29645:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 788.054862] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 791.741652] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 799.254320] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 804.504957] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 810.457105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 810.716565] Lustre: 31612:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 811.980046] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 812.032976] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 812.080547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 812.091280] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 816.604744] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 826.068028] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 832.658570] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 852.966280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 852.967873] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 852.980811] LustreError: Skipped 1 previous similar message [ 852.982139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 853.011337] Lustre: Skipped 1 previous similar message [ 866.271636] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 866.535329] Lustre: server umount lustre-OST0000 complete [ 876.553398] Lustre: server umount lustre-MDT0000 complete [ 878.561244] LustreError: 30447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 878.574055] LustreError: 30447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 880.011665] LustreError: 30437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437830 with bad export cookie 16756819319782117679 [ 880.016525] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 880.025419] LustreError: 30437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 880.348785] Lustre: server umount lustre-MDT0001 complete [ 889.964668] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [ 892.482696] Key type lgssc unregistered [ 892.732514] LNet: 33598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.736424] LNetError: 33598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 892.750788] LNet: Removed LNI 192.168.206.137@tcp [ 893.523143] Key type .llcrypt unregistered [ 893.527269] Key type ._llcrypt unregistered [ 914.069253] Key type ._llcrypt registered [ 914.072720] Key type .llcrypt registered [ 914.177280] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 914.936918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 914.955281] alg: No test for adler32 (adler32-zlib) [ 915.929742] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 916.180474] LNet: Added LNI 192.168.206.137@tcp [8/256/0/180] [ 917.847200] Key type lgssc registered [ 918.887892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 925.906607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 925.923454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 927.284217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 931.386973] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 936.417419] LustreError: 35003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 936.437606] LustreError: 35003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 939.177225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 939.510453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 943.322695] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 950.074762] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 955.067723] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 961.777979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 962.187415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 965.226820] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 967.880917] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 972.438519] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 975.362969] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 990.558896] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 08:18:59 (1788437939) [ 992.754533] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 994.908261] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1001.942959] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 08:19:11 (1788437951) [ 1011.039128] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 08:19:20 (1788437960) [ 1012.075409] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1020.563439] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1020.570924] Lustre: Skipped 2 previous similar messages [ 1022.442768] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 08:19:31 (1788437971) [ 1025.013202] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1025.016050] Lustre: Skipped 2 previous similar messages [ 1033.224644] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1033.229269] Lustre: Skipped 4 previous similar messages [ 1047.729569] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 08:19:56 (1788437996) [ 1050.410858] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1054.122506] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1065.071518] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 08:20:14 (1788438014) [ 1067.218696] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1067.223637] Lustre: Skipped 6 previous similar messages [ 1081.436927] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 08:20:30 (1788438030) [ 1098.721583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1098.739377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1098.751528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1100.772905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1100.773124] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1100.780309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1104.661529] Lustre: server umount lustre-OST0000 complete [ 1111.009144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1111.011920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1111.021574] Lustre: Skipped 1 previous similar message [ 1111.032120] Lustre: Skipped 2 previous similar messages [ 1114.903476] Lustre: server umount lustre-MDT0000 complete [ 1116.129849] LustreError: 35002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1116.151338] LustreError: 35002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1118.200105] LustreError: 34982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438068 with bad export cookie 14699787604735687238 [ 1118.213654] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1118.548300] Lustre: server umount lustre-MDT0001 complete [ 1128.501890] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [ 1131.220163] Key type lgssc unregistered [ 1131.498829] LNet: 42865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1131.508492] LNetError: 42865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1131.529924] LNet: Removed LNI 192.168.206.137@tcp [ 1132.347990] Key type .llcrypt unregistered [ 1132.349883] Key type ._llcrypt unregistered [ 1149.463678] Key type ._llcrypt registered [ 1149.471540] Key type .llcrypt registered [ 1149.617579] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_hostid [ 1165.289028] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 1166.552879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1166.593525] alg: No test for adler32 (adler32-zlib) [ 1167.732317] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1168.053050] LNet: Added LNI 192.168.206.137@tcp [8/256/0/180] [ 1169.735591] Key type lgssc registered [ 1171.083772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1207.511419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1207.551169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1208.872276] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1208.894699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1208.954501] Lustre: lustre-MDT0000: new disk, initializing [ 1209.004354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1209.020186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1213.403891] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 1223.125490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1223.245461] Lustre: 46015:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1223.303757] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1223.306918] Lustre: Skipped 1 previous similar message [ 1223.374576] Lustre: lustre-MDT0001: new disk, initializing [ 1223.477618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1223.543148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1223.561191] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1227.717294] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 1237.250765] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1242.710975] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1250.443219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1250.762870] Lustre: lustre-OST0000: new disk, initializing [ 1250.768184] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1250.777441] Lustre: 47984:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1250.878532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1257.023610] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 1261.081936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1261.095828] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1261.237049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1266.305536] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1271.117628] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1271.311453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1275.798983] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1275.968961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1281.506741] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1281.519944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1281.540995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1282.016521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1282.026191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1287.146405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1287.156708] Lustre: Skipped 3 previous similar messages [ 1292.255933] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1292.264741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1292.281148] Lustre: Skipped 1 previous similar message [ 1292.417147] Lustre: server umount lustre-OST0000 complete [ 1297.387220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1297.396593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1297.402275] Lustre: Skipped 1 previous similar message [ 1302.498161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1302.509205] Lustre: Skipped 2 previous similar messages [ 1303.000784] Lustre: server umount lustre-MDT0000 complete [ 1306.605488] LustreError: 46008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438257 with bad export cookie 1692289016775898857 [ 1306.610644] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1306.618099] LustreError: 46008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1306.964194] Lustre: server umount lustre-MDT0001 complete [ 1316.605625] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 08:24:25 (1788438265) [ 1322.713858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1323.184426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1326.879464] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 1328.615831] LustreError: 50448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1328.629338] LustreError: 50448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1332.708144] LustreError: 50447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1335.942958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1336.272912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1340.684859] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 1348.048437] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1353.146941] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1359.017237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1359.319580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1364.867429] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 1371.527106] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1375.985890] LustreError: 52975:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1376.895928] Lustre: Modifying parameter general.timeout=1 in log params [ 1378.537432] Lustre: Modifying parameter general.timeout=3 in log params [ 1378.541207] Lustre: Skipped 1 previous similar message [ 1381.053609] Lustre: Modifying parameter general.timeout=6 in log params [ 1381.063216] Lustre: Skipped 2 previous similar messages [ 1385.476387] Lustre: Modifying parameter general.timeout=11 in log params [ 1385.489903] Lustre: Skipped 4 previous similar messages [ 1393.518783] Lustre: Modifying parameter general.timeout=20 in log params [ 1393.532604] Lustre: Skipped 8 previous similar messages [ 1409.524905] Lustre: Modifying parameter general.timeout=36 in log params [ 1409.534559] Lustre: Skipped 15 previous similar messages [ 1441.564350] Lustre: Modifying parameter general.timeout=72 in log params [ 1441.569383] Lustre: Skipped 35 previous similar messages [ 1499.344537] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 08:27:28 (1788438448) [ 1502.683764] Lustre: 59919:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1504.406597] Lustre: 60017:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1507.237917] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 in log params [ 1507.251052] Lustre: Skipped 62 previous similar messages [ 1508.266742] Lustre: 60228:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1508.275242] Lustre: 60228:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1537.866481] Lustre: 61797:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1537.873503] Lustre: 61797:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1537.893349] LustreError: 61797:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1542.526342] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 08:28:11 (1788438491) [ 1556.455987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1556.471454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1556.492860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1561.061704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1562.705637] Lustre: server umount lustre-MDT0000 complete [ 1566.177412] LustreError: 51175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1566.212749] LustreError: 51175:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1566.289855] LustreError: 50427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438516 with bad export cookie 1692289016775899739 [ 1566.299084] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1566.305180] LustreError: 50427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1566.648787] Lustre: server umount lustre-MDT0001 complete [ 1580.864197] Lustre: server umount lustre-OST0000 complete [ 1605.060703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1605.367740] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1605.633735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1610.305446] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 1619.252974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1619.385454] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1619.446365] Lustre: 64329:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1624.375653] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 1631.668375] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1637.570205] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1644.376142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1644.555781] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1644.740126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1644.745531] Lustre: Skipped 1 previous similar message [ 1650.505907] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 1657.188970] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1659.136486] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1659.143636] Lustre: Skipped 18 previous similar messages [ 1659.179047] Lustre: 66900:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1670.644116] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1670.648649] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1670.667920] Lustre: Skipped 1 previous similar message [ 1670.676317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1670.676325] Lustre: Skipped 3 previous similar messages [ 1672.922783] Lustre: server umount lustre-OST0000 complete [ 1682.879383] Lustre: server umount lustre-MDT0000 complete [ 1685.984464] LustreError: 64334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1686.021248] LustreError: 64334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1687.060314] LustreError: 64319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438637 with bad export cookie 1692289016775902847 [ 1687.064463] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1687.108752] LustreError: 64319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1687.604894] Lustre: server umount lustre-MDT0001 complete [ 1701.013483] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [ 1703.875489] Key type lgssc unregistered [ 1704.174749] LNet: 68109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1704.179647] LNetError: 68109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1704.192767] LNet: Removed LNI 192.168.206.137@tcp [ 1705.183438] Key type .llcrypt unregistered [ 1705.185229] Key type ._llcrypt unregistered [ 1729.633313] Key type ._llcrypt registered [ 1729.635849] Key type .llcrypt registered [ 1729.819349] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 1730.980837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1731.041600] alg: No test for adler32 (adler32-zlib) [ 1732.239800] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1732.635524] LNet: Added LNI 192.168.206.137@tcp [8/256/0/180] [ 1734.335886] Key type lgssc registered [ 1735.225861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1742.874669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1742.897215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1744.569571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1748.911286] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 1755.109124] LustreError: 69520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1755.147788] LustreError: 69520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1758.422721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1758.815707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1763.037542] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 1770.850135] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1776.956216] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1784.204192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1784.819966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1790.488485] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 1797.802477] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1809.033616] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1819.618753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1819.624604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1819.631771] Lustre: Skipped 2 previous similar messages [ 1819.651706] Lustre: Skipped 2 previous similar messages [ 1824.741625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1824.750367] Lustre: Skipped 2 previous similar messages [ 1829.859857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1829.872616] Lustre: Skipped 2 previous similar messages [ 1831.391260] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1831.555404] Lustre: server umount lustre-MDT0000 complete [ 1834.982055] LustreError: 69516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1835.009380] LustreError: 69516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1835.363731] LustreError: 69500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438785 with bad export cookie 15475498076582343162 [ 1835.373196] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1835.379797] LustreError: 69500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1835.831088] Lustre: server umount lustre-MDT0001 complete [ 1850.476665] Lustre: server umount lustre-OST0000 complete [ 1877.812980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1878.080106] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1878.130822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1878.393360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1883.067636] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 1891.004206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1891.096772] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1891.135474] Lustre: 74389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1891.540448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1895.743625] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 1903.514025] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1909.152123] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1915.392105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1915.589758] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1915.905236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1922.043409] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 1928.350334] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1929.845687] Lustre: 76968:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1931.507362] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1938.401238] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1938.413034] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1938.425257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1938.914429] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1941.951700] Lustre: server umount lustre-OST0000 complete [ 1947.616084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1947.623062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1947.639439] Lustre: Skipped 1 previous similar message [ 1947.647913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1947.659733] Lustre: Skipped 1 previous similar message [ 1951.788982] Lustre: server umount lustre-MDT0000 complete [ 1954.274597] LustreError: 75142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1954.291741] LustreError: 75142:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1955.225979] LustreError: 75150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438905 with bad export cookie 15475498076582344009 [ 1955.231922] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1955.236044] LustreError: 75150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1955.674983] Lustre: server umount lustre-MDT0001 complete [ 1967.017687] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [ 1969.887528] Key type lgssc unregistered [ 1970.169985] LNet: 78182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1970.179842] LNetError: 78182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1970.201038] LNet: Removed LNI 192.168.206.137@tcp [ 1970.893831] Key type .llcrypt unregistered [ 1970.896247] Key type ._llcrypt unregistered [ 1998.007290] Key type ._llcrypt registered [ 1998.009476] Key type .llcrypt registered [ 1998.105438] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 1999.464682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1999.570388] alg: No test for adler32 (adler32-zlib) [ 2000.706840] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2001.003912] LNet: Added LNI 192.168.206.137@tcp [8/256/0/180] [ 2002.743210] Key type lgssc registered [ 2003.989564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2012.277381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2012.290463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2013.892049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2018.573384] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 2023.421962] LustreError: 79592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2023.451710] LustreError: 79592:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2028.526557] LustreError: 79591:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2028.963421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2029.889146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2034.661350] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 2043.990035] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2050.210941] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2058.024789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2058.726354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2065.362422] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 2074.712316] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2082.798505] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2084.406840] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2085.653352] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2088.044466] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2088.050185] Lustre: Skipped 1 previous similar message [ 2092.598985] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2092.604833] Lustre: Skipped 3 previous similar messages [ 2100.766256] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2100.772463] Lustre: Skipped 7 previous similar messages [ 2117.854967] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2117.863330] Lustre: Skipped 14 previous similar messages [ 2150.173775] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2150.181258] Lustre: Skipped 33 previous similar messages [ 2191.195585] Lustre: 87697:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2194.173938] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 08:39:03 (1788439143) [ 2195.939223] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2198.065515] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 08:39:07 (1788439147) [ 2203.105137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2203.113455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2203.130982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2207.203719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2207.204395] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2207.208384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2207.212740] Lustre: Skipped 1 previous similar message [ 2208.385318] Lustre: server umount lustre-OST0000 complete [ 2212.321225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2212.355272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2212.360281] Lustre: Skipped 1 previous similar message [ 2217.446819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2217.456248] Lustre: Skipped 1 previous similar message [ 2218.590538] Lustre: server umount lustre-MDT0000 complete [ 2222.166669] LustreError: 79573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439172 with bad export cookie 2820905223982000078 [ 2222.174628] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2222.188297] LustreError: 79573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2222.499322] Lustre: server umount lustre-MDT0001 complete [ 2234.629521] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [ 2238.082035] Key type lgssc unregistered [ 2238.543712] LNet: 88946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2238.554062] LNetError: 88946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2238.567866] LNet: Removed LNI 192.168.206.137@tcp [ 2239.800424] Key type .llcrypt unregistered [ 2239.803644] Key type ._llcrypt unregistered [ 2245.259082] Key type ._llcrypt registered [ 2245.263963] Key type .llcrypt registered [ 2245.453814] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2250.394954] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules [ 2251.209845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2251.368398] alg: No test for adler32 (adler32-zlib) [ 2252.372202] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2252.581935] LNet: Added LNI 192.168.206.137@tcp [8/256/0/180] [ 2254.216126] Key type lgssc registered [ 2255.382884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2256.867478] LustreError: 90144:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2274.202913] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 2278.908398] LustreError: 90144:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2284.960955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2284.982829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2286.440654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2291.400054] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 2296.825684] LustreError: 91169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2296.839338] LustreError: 91169:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2300.979339] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 08:40:50 (1788439250) [ 2301.921040] LustreError: 91170:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2306.201285] LustreError: 91201:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2306.220713] LustreError: 91201:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2306.229795] LustreError: 91201:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2306.621928] Lustre: server umount lustre-MDT0000 complete [ 2321.020297] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [ 2324.157916] Key type lgssc unregistered [ 2324.442763] LNet: 92695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2324.449246] LNetError: 92695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2324.464961] LNet: Removed LNI 192.168.206.137@tcp [ 2325.350242] Key type .llcrypt unregistered [ 2325.352856] Key type ._llcrypt unregistered [ 2342.654558] Key type ._llcrypt registered [ 2342.657688] Key type .llcrypt registered [ 2342.763514] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 2344.047594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2344.102876] alg: No test for adler32 (adler32-zlib) [ 2345.089826] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2345.288778] LNet: Added LNI 192.168.206.137@tcp [8/256/0/180] [ 2346.935221] Key type lgssc registered [ 2348.059110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2355.888977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2355.906560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2357.493557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2362.075557] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 2366.948152] LustreError: 93914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.966781] LustreError: 93914:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2371.682483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2372.325329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2376.501957] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 2384.516155] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2390.937344] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2399.354676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2399.999372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2406.843864] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 2414.847034] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2469.282211] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 08:43:38 (1788439418) [ 2471.079663] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2472.904426] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 08:43:42 (1788439422) [ 2479.074626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2479.082991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2479.101412] Lustre: Skipped 2 previous similar messages [ 2479.111698] Lustre: Skipped 2 previous similar messages [ 2483.607555] Lustre: server umount lustre-MDT0000 complete [ 2484.200203] LustreError: 96706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2484.229834] LustreError: 96706:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2487.397565] LustreError: 93894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439438 with bad export cookie 17301106285696180257 [ 2487.403092] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2487.410557] LustreError: 93894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2487.644119] Lustre: server umount lustre-MDT0001 complete [ 2502.423643] Lustre: server umount lustre-OST0000 complete [ 2515.505348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2516.028974] LustreError: 98212:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2516.225442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2521.238604] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 2521.567836] LustreError: 98213:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2525.689274] LustreError: 98212:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2531.145466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2531.808290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2536.420773] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 2544.834844] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2552.531413] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2567.553083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2567.806534] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2567.826549] LustreError: 98201:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2567.875684] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2567.885523] LustreError: 100297:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2567.893727] LustreError: 100297:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2567.900553] LustreError: 100297:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2568.031531] Lustre: server umount lustre-OST0000 complete [ 2568.035035] LustreError: 100297:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2577.136705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2577.318387] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2577.325611] LustreError: 98201:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2577.348317] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2577.356479] LustreError: 100765:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2577.362700] LustreError: 100765:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2577.370337] LustreError: 100765:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2577.454248] Lustre: server umount lustre-OST0000 complete [ 2577.456436] LustreError: 100765:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2590.007392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2590.201737] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2590.211596] Lustre: Found index 0 for lustre-OST0000, updating log [ 2590.217234] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2590.297987] Lustre: lustre-OST0000: new disk, initializing [ 2590.301502] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2590.309483] Lustre: 101299:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2590.660918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2597.829897] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 2601.962415] Lustre: Failing over lustre-OST0000 [ 2602.126610] Lustre: server umount lustre-OST0000 complete [ 2607.073316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2607.075269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2607.087896] Lustre: Skipped 1 previous similar message [ 2607.109027] Lustre: Skipped 1 previous similar message [ 2612.193284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2612.203926] Lustre: Skipped 1 previous similar message [ 2612.584119] Lustre: server umount lustre-MDT0000 complete [ 2616.894813] LustreError: 98192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439567 with bad export cookie 17301106285696183001 [ 2616.910376] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2616.915792] LustreError: 98192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2626.697300] Lustre: server umount lustre-MDT0001 complete [ 2636.711410] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 08:46:26 (1788439586) [ 2654.204424] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 2666.292660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2666.868954] LustreError: 103646:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2666.900106] LustreError: 103646:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2667.058242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2671.673300] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 2680.914908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2686.195843] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 2689.454207] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2695.875865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2696.922390] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2703.291550] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 2706.086052] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2706.109986] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2706.114350] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2712.125909] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2712.251211] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2712.373817] Lustre: lustre-OST0001: new disk, initializing [ 2712.378563] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2712.383590] Lustre: 105966:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2712.638066] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2712.649843] Lustre: Skipped 2 previous similar messages [ 2719.837504] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 2721.459492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2721.468508] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2721.629316] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2729.796134] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2738.837475] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2747.376309] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2747.398514] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2747.404878] Lustre: Skipped 2 previous similar messages [ 2747.426085] Lustre: Skipped 2 previous similar messages [ 2750.038579] Lustre: server umount lustre-MDT0001 complete [ 2752.484590] LustreError: 104378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2752.495052] LustreError: 104378:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2760.046708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2760.203796] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2760.209409] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2760.275925] Lustre: 103635:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2762.347064] LustreError: 107548:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2762.357536] LustreError: 107548:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2762.364755] LustreError: 103638:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2762.432421] LustreError: 107554:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.137@tcp: cfg command failed: rc = -17 [ 2762.436827] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2762.436827] [ 2765.481680] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 2766.317295] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2775.384606] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 08:48:44 (1788439724) [ 2794.593416] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 2813.925469] Lustre: 109582:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2840.854940] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2845.466698] Lustre: 110717:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2939.872366] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2939.881287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2939.904059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2939.910369] Lustre: Skipped 1 previous similar message [ 2945.687306] Lustre: server umount lustre-MDT0000 complete [ 2950.106182] LustreError: 106687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439900 with bad export cookie 17301106285696184002 [ 2950.108127] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2950.123349] LustreError: 106687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2950.644078] Lustre: server umount lustre-MDT0001 complete [ 2965.462525] Lustre: server umount lustre-OST0000 complete [ 2967.014168] Lustre: 93224:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788439901/real 1788439901] req@ffff98bfc69b5880 x1875314522563968/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788439917 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2967.056565] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2967.071380] Lustre: Skipped 3 previous similar messages [ 2970.435449] Lustre: server umount lustre-OST0001 complete [ 2976.733922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2988.715165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2998.026563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3008.337895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3030.026411] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 3041.791510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3041.825437] Lustre: lustre-MDT0000: reset Object Index mappings [ 3042.503520] LustreError: 114669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3042.536545] LustreError: 114669:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 3042.762681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3042.775266] Lustre: Skipped 1 previous similar message [ 3048.342520] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 3059.265081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3059.281681] Lustre: lustre-MDT0001: reset Object Index mappings [ 3064.737304] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 3067.657850] Lustre: 115818:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3074.881424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3081.777756] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 3084.985460] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3089.144035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3091.029749] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3097.086064] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3097.108089] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3098.215607] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 3106.836264] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3110.452554] Lustre: 117694:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3168.461471] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 08:55:17 (1788440117) [ 3194.344963] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3194.367198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3194.388937] Lustre: Skipped 2 previous similar messages [ 3194.410418] Lustre: Skipped 11 previous similar messages [ 3199.672103] Lustre: server umount lustre-MDT0000 complete [ 3203.090226] LustreError: 114650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440153 with bad export cookie 17301106285696230874 [ 3203.100241] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3203.118290] LustreError: 114650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3204.580311] LustreError: 114666:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3204.613927] LustreError: 114666:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3209.668349] Lustre: server umount lustre-MDT0001 complete [ 3220.132895] Lustre: server umount lustre-OST0000 complete [ 3224.082580] LustreError: 118657:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000401:0x35:0x0].0x0 (ffff98bfc4441600) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3238.367216] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3238.581882] Lustre: server umount lustre-OST0001 complete [ 3245.917217] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_hostid [ 3257.246466] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 3295.419227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3295.708193] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3295.743714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3295.856238] Lustre: lustre-MDT0000: new disk, initializing [ 3295.933528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3295.938881] Lustre: Skipped 3 previous similar messages [ 3295.968918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3300.140231] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 3307.582485] Lustre: server umount lustre-MDT0000 complete [ 3317.980774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3318.417717] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3318.433623] Lustre: 121968:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3318.446363] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3322.701739] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 3327.862750] Lustre: server umount lustre-MDT0000 complete [ 3337.769385] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 08:58:07 (1788440287) [ 3339.585934] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3341.376430] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 08:58:10 (1788440290) [ 3360.211225] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 3370.232881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3375.461822] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 3384.020759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3384.116333] Lustre: 124017:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3384.161923] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3384.166936] Lustre: Skipped 1 previous similar message [ 3384.244909] Lustre: lustre-MDT0001: new disk, initializing [ 3384.340351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3384.355069] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3388.692483] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 3393.544105] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3400.133271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3400.347621] Lustre: lustre-OST0000: new disk, initializing [ 3400.351380] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3400.357491] Lustre: 125597:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3402.392638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3402.403906] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3402.520145] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3406.315404] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 3418.385081] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3418.651809] Lustre: lustre-OST0001: new disk, initializing [ 3418.656258] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3418.661701] Lustre: 126467:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3420.046416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3420.058875] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3420.149845] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3424.683446] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 3434.643803] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3438.731454] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3846.910847] Lustre: 132897:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3846.918874] Lustre: 132897:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3846.922264] Lustre: 132897:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3846.936364] Lustre: 132897:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3846.948273] Lustre: 132897:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3846.957808] Lustre: 132897:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4162.021927] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:11:50 (1788441110) [ 4177.893062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4177.897268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4177.913221] Lustre: Skipped 4 previous similar messages [ 4177.921961] Lustre: Skipped 8 previous similar messages [ 4183.011415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4183.020317] Lustre: Skipped 6 previous similar messages [ 4183.756525] Lustre: server umount lustre-MDT0000 complete [ 4187.662469] LustreError: 124756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441138 with bad export cookie 17301106285696301000 [ 4187.664962] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4187.670626] LustreError: 124756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4188.129245] LustreError: 127272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4188.131518] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4188.132451] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4188.146042] LustreError: 127272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4188.187061] Lustre: Skipped 3 previous similar messages [ 4194.401745] Lustre: server umount lustre-MDT0001 complete [ 4204.655548] Lustre: server umount lustre-OST0000 complete [ 4215.720857] Lustre: server umount lustre-OST0001 complete [ 4223.758982] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_hostid [ 4234.224391] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 4277.832635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4278.053807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4278.079766] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4278.140331] Lustre: lustre-MDT0000: new disk, initializing [ 4278.196775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4278.202284] Lustre: Skipped 5 previous similar messages [ 4278.222976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4283.481266] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 4294.371635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4294.585650] Lustre: 135960:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4294.645716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4294.649838] Lustre: Skipped 1 previous similar message [ 4294.753914] Lustre: lustre-MDT0001: new disk, initializing [ 4294.926424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4294.936193] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4299.183281] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 4308.353830] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4313.818389] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4320.072173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4320.323751] Lustre: lustre-OST0000: new disk, initializing [ 4320.329768] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4320.334336] Lustre: 137927:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4320.412844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4320.419931] Lustre: Skipped 1 previous similar message [ 4321.600170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4321.606833] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4321.732824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4325.616267] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 4334.492865] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4344.762637] Lustre: lustre-MDD0000: changelog on [ 4347.678335] Lustre: lustre-MDD0001: changelog on [ 6245.125175] Lustre: lustre-MDD0001: changelog off [ 6246.863501] Lustre: 144854:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6247.718641] Lustre: lustre-MDD0000: changelog off [ 6256.608094] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6256.619537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6256.640296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6256.649587] Lustre: Skipped 3 previous similar messages [ 6257.121028] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6257.133260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6262.241802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6262.246891] Lustre: Skipped 3 previous similar messages [ 6267.363902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6267.373335] Lustre: Skipped 1 previous similar message [ 6267.871445] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6268.139456] Lustre: server umount lustre-OST0000 complete [ 6272.487646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6272.491254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6272.507399] Lustre: Skipped 1 previous similar message [ 6272.520622] Lustre: Skipped 1 previous similar message [ 6278.096668] Lustre: server umount lustre-MDT0000 complete [ 6281.454931] LustreError: 135950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443232 with bad export cookie 17301106285697121596 [ 6281.463943] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6281.466066] LustreError: 135950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6281.857654] Lustre: server umount lustre-MDT0001 complete [ 6291.465565] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [ 6293.723500] Key type lgssc unregistered [ 6294.040928] LNet: 147062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6294.045891] LNetError: 147062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6295.091973] LNet: Removed LNI 192.168.206.137@tcp [ 6295.899187] Key type .llcrypt unregistered [ 6295.900988] Key type ._llcrypt unregistered [ 6313.624107] Key type ._llcrypt registered [ 6313.626302] Key type .llcrypt registered [ 6313.741866] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_hostid [ 6329.007196] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 6329.955945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6329.992708] alg: No test for adler32 (adler32-zlib) [ 6331.027585] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6331.230125] LNet: Added LNI 192.168.206.137@tcp [8/256/0/180] [ 6332.871594] Key type lgssc registered [ 6333.926370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6368.107447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6368.145810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6369.421925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6369.448111] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6369.556253] Lustre: lustre-MDT0000: new disk, initializing [ 6369.652409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6369.672567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6374.713458] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 6385.512055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6385.624874] Lustre: 150259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6385.688917] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6385.693030] Lustre: Skipped 1 previous similar message [ 6385.751476] Lustre: lustre-MDT0001: new disk, initializing [ 6385.830197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6385.868464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6385.873602] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6390.404690] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 6400.629962] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6406.009876] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6412.481782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6412.720869] Lustre: lustre-OST0000: new disk, initializing [ 6412.724801] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6412.732164] Lustre: 152237:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6412.836763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6418.849613] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 6422.559596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6422.570912] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6422.682518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6428.218118] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6433.610452] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_module obdecho/obdecho [ 6435.496664] LustreError: 153240:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6435.511938] LustreError: 153240:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6437.073821] LustreError: 153336:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6448.608893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6448.613846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6448.630704] Lustre: Skipped 1 previous similar message [ 6448.645301] Lustre: Skipped 2 previous similar messages [ 6453.730301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6454.666499] Lustre: server umount lustre-MDT0000 complete [ 6458.131356] LustreError: 150252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443408 with bad export cookie 6679445939031537132 [ 6458.132811] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6458.160842] LustreError: 150252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6458.513780] Lustre: server umount lustre-MDT0001 complete [ 6471.605284] Lustre: server umount lustre-OST0000 complete [ 6493.164307] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_hostid [ 6504.477686] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 6539.844671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6540.032655] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6540.069748] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6540.142734] Lustre: lustre-MDT0000: new disk, initializing [ 6540.228546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6540.244662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6544.666444] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 6553.645466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6553.801747] Lustre: 157052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6553.897718] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6553.904595] Lustre: Skipped 1 previous similar message [ 6554.014800] Lustre: lustre-MDT0001: new disk, initializing [ 6554.115478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6554.154911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6554.170433] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6558.925252] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 6568.876249] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6574.730358] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6581.447034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6581.783871] Lustre: lustre-OST0000: new disk, initializing [ 6581.787237] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6581.794266] Lustre: 159021:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6581.869847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6588.488487] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 6592.120628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6592.130815] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6592.264705] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6598.755414] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6604.942326] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6605.178623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6610.096965] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6610.284310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6612.966667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6612.969493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6612.978327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6612.978337] Lustre: Skipped 2 previous similar messages [ 6613.000237] Lustre: Skipped 2 previous similar messages [ 6613.007951] LustreError: Skipped 1 previous similar message [ 6618.086127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6618.091642] Lustre: Skipped 1 previous similar message [ 6618.658800] Lustre: server umount lustre-OST0000 complete [ 6623.210116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6623.217214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6623.221407] Lustre: Skipped 1 previous similar message [ 6623.243027] Lustre: Skipped 2 previous similar messages [ 6628.938181] Lustre: server umount lustre-MDT0000 complete [ 6632.632810] LustreError: 157046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443583 with bad export cookie 6679445939031538091 [ 6632.642510] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6632.648171] LustreError: 157046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6633.013916] Lustre: server umount lustre-MDT0001 complete [ 6636.546231] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 09:53:06 (1788443586) [ 6638.339393] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6640.191736] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 09:53:09 (1788443589) [ 6647.641273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6648.166211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6653.101979] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 6653.419672] LustreError: 161432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6653.450634] LustreError: 161432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6657.517181] LustreError: 161433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6662.268444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6662.625777] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6662.646137] Lustre: Skipped 2 previous similar messages [ 6668.424823] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 6676.968807] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6684.307228] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6691.317997] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6691.649894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6691.657638] Lustre: Skipped 1 previous similar message [ 6697.957934] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 6705.863323] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6718.944849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6718.956391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6718.984349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6725.183859] Lustre: server umount lustre-MDT0000 complete [ 6727.135686] LustreError: 161429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6729.704826] LustreError: 162402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6729.735270] LustreError: 162402:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6729.772844] LustreError: 162167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443680 with bad export cookie 6679445939031538966 [ 6729.789935] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6729.802867] LustreError: 162167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6734.818956] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6734.820051] LustreError: 161427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6734.855714] Lustre: Skipped 2 previous similar messages [ 6734.882777] LustreError: 161427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6736.384718] Lustre: server umount lustre-MDT0001 complete [ 6743.367693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6757.425596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6757.620045] LustreError: 165295:0:(llog.c:1655:llog_backup()) MGC192.168.206.137@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6757.630135] Lustre: 165295:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.137@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6763.027531] LustreError: 165300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6779.346833] LustreError: 165301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6779.365605] LustreError: 165301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6811.616871] LustreError: 165302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6811.631873] LustreError: 165302:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 25 previous similar messages [ 6818.271307] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6818.276395] Lustre: 165295:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6859.261272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6864.129910] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 6864.365645] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6872.845318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6875.626256] LustreError: 165319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6875.642667] LustreError: 165319:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 45 previous similar messages [ 6880.145168] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 6881.268069] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6886.958562] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6888.268771] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6892.835939] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6894.031804] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6911.972026] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6911.976757] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6911.983909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6912.013729] Lustre: Skipped 11 previous similar messages [ 6917.796856] Lustre: server umount lustre-MDT0000 complete [ 6921.676458] LustreError: 161412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443872 with bad export cookie 6679445939031545427 [ 6921.679655] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6921.692497] LustreError: 161412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6922.071139] Lustre: server umount lustre-MDT0001 complete [ 6936.613656] Lustre: server umount lustre-OST0000 complete [ 6956.580275] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_hostid [ 6966.040604] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 7000.793847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7001.006743] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7001.027529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7001.167646] Lustre: lustre-MDT0000: new disk, initializing [ 7001.232924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7001.239292] Lustre: Skipped 1 previous similar message [ 7001.252217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7006.114365] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 7019.668963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7019.871888] Lustre: 170400:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7020.209233] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7025.830375] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 7038.199338] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7046.678330] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7055.099761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7055.384727] Lustre: lustre-OST0000: new disk, initializing [ 7055.388531] Lustre: Skipped 1 previous similar message [ 7055.392167] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7055.397275] Lustre: Skipped 2 previous similar messages [ 7055.403170] Lustre: 172370:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7061.425987] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 7063.599047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7063.610548] Lustre: Skipped 1 previous similar message [ 7063.613372] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7063.713956] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7070.616756] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7075.294611] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7075.512215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7080.102962] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7080.455574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7084.530991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7084.537335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7084.554857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7084.569164] Lustre: Skipped 3 previous similar messages [ 7084.584600] Lustre: Skipped 4 previous similar messages [ 7088.717617] Lustre: server umount lustre-OST0000 complete [ 7098.919123] Lustre: server umount lustre-MDT0000 complete [ 7099.874311] LustreError: 170412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7099.889565] LustreError: 170412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7103.122493] LustreError: 171195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444053 with bad export cookie 6679445939031562262 [ 7103.125016] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7103.134728] LustreError: 171195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7103.737578] Lustre: server umount lustre-MDT0001 complete [ 7114.550849] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 10:01:03 (1788444063) [ 7121.351920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7125.787876] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 7133.657083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7137.645217] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 7144.142181] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7150.630361] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7159.362586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7168.405908] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 7175.323800] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7187.622636] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 10:02:17 (1788444137) [ 7193.056679] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7193.065341] LustreError: Skipped 1 previous similar message [ 7193.069944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7193.077371] Lustre: Skipped 2 previous similar messages [ 7197.576465] Lustre: server umount lustre-OST0000 complete [ 7213.544300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7213.554451] Lustre: Skipped 11 previous similar messages [ 7215.074141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7215.250605] Lustre: server umount lustre-MDT0000 complete [ 7219.969839] LustreError: 174772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444170 with bad export cookie 6679445939031563144 [ 7219.984228] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7219.996167] LustreError: 174772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7220.393023] Lustre: server umount lustre-MDT0001 complete [ 7230.809203] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [ 7234.049887] Key type lgssc unregistered [ 7234.374550] LNet: 178521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7234.399365] LNetError: 178521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7234.416425] LNet: Removed LNI 192.168.206.137@tcp [ 7235.439933] Key type .llcrypt unregistered [ 7235.442895] Key type ._llcrypt unregistered [ 7254.799229] Key type ._llcrypt registered [ 7254.809897] Key type .llcrypt registered [ 7255.016280] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 7256.046039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7256.075144] alg: No test for adler32 (adler32-zlib) [ 7257.127384] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7257.418602] LNet: Added LNI 192.168.206.137@tcp [8/256/0/180] [ 7259.135409] Key type lgssc registered [ 7260.308395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7268.612703] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7268.666417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7284.320472] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7289.439268] LustreError: 179760:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.137@tcp: failed processing log, type 4: rc = -110 [ 7294.624424] LustreError: 179760:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7294.654541] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7294.674154] LustreError: MGC192.168.206.137@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7294.690324] LustreError: 179760:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7294.704249] LustreError: 179760:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 7294.722834] LustreError: 179760:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7294.831367] Lustre: server umount lustre-OST0000 complete [ 7294.833315] LustreError: 179760:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7304.024038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7305.534489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7309.869701] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 7315.944357] LustreError: 180261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7315.967343] LustreError: 180261:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7319.386727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7319.912502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7325.304440] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 7334.233471] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7340.403311] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7347.498605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7347.812518] LustreError: 182077:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7347.827207] Lustre: 182077:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.137@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7348.040637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7354.497525] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 7361.874897] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7364.551856] LustreError: lustre-MDT0000-lwp-MDT0001: operation fld_query to node 0@lo failed: rc = -107 [ 7364.557445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7364.575399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7366.116844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7366.129619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7366.148900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7369.697738] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7369.710719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7369.734963] Lustre: Skipped 2 previous similar messages [ 7371.386241] Lustre: server umount lustre-MDT0000 complete [ 7374.837870] LustreError: 182837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7374.856341] LustreError: 182837:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7376.518907] LustreError: 180994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444327 with bad export cookie 16663537422505391391 [ 7376.520467] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7376.541445] LustreError: 180994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7376.841344] LustreError: 180989:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e5e3139f x1875319673878656/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7377.044361] Lustre: server umount lustre-MDT0001 complete [ 7394.306498] Lustre: server umount lustre-OST0000 complete [ 7404.677848] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_hostid [ 7415.777832] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 7453.159314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7453.342874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7453.364719] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7453.423599] Lustre: lustre-MDT0000: new disk, initializing [ 7453.505467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7453.527680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7458.595400] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 7471.279527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7471.404892] Lustre: 185602:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7471.475940] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7471.480862] Lustre: Skipped 1 previous similar message [ 7471.594395] Lustre: lustre-MDT0001: new disk, initializing [ 7471.683711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7471.722146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7471.741237] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7477.275901] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 7486.551823] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7492.131893] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7498.723122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7498.972295] Lustre: lustre-OST0000: new disk, initializing [ 7498.975815] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7498.985988] Lustre: 187572:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7499.077704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7504.685786] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 7507.517614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7507.531473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7507.633665] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7514.864336] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7520.121231] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7520.348783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7524.729488] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7524.899386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7528.416338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7528.422286] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7528.434634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7528.441932] Lustre: Skipped 1 previous similar message [ 7532.983991] Lustre: server umount lustre-OST0000 complete [ 7538.145497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7538.156088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7538.170952] Lustre: Skipped 1 previous similar message [ 7538.184172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7543.126113] Lustre: server umount lustre-MDT0000 complete [ 7543.781822] LustreError: 185609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7543.798905] LustreError: 185609:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7546.165158] LustreError: 185593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444496 with bad export cookie 16663537422505392105 [ 7546.171411] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7546.175241] LustreError: 185593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7546.466926] Lustre: server umount lustre-MDT0001 complete [ 7555.039609] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 10:08:24 (1788444504) [ 7573.223182] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [ 7575.576835] Key type lgssc unregistered [ 7575.842676] LNet: 190514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7575.852934] LNetError: 190514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7575.867170] LNet: Removed LNI 192.168.206.137@tcp [ 7576.568773] Key type .llcrypt unregistered [ 7576.571566] Key type ._llcrypt unregistered [ 7592.278366] Key type ._llcrypt registered [ 7592.280627] Key type .llcrypt registered [ 7592.381597] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 7593.205677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7593.297662] alg: No test for adler32 (adler32-zlib) [ 7594.273222] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7594.454560] LNet: Added LNI 192.168.206.137@tcp [8/256/0/180] [ 7596.103219] Key type lgssc registered [ 7597.242376] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7604.556475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7604.591516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7606.019611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7610.099889] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 7615.462453] LustreError: 191730:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7615.491611] LustreError: 191730:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7618.351149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7618.836093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7623.370290] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 7634.688524] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7641.389402] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7650.804082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7651.102477] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7651.104658] LustreError: 193598:0:(llog.c:1696:llog_backup()) MGC192.168.206.137@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7651.122449] Lustre: 193598:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.137@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7651.291785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7657.619651] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 7665.461980] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7668.707229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7668.713747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7668.732936] Lustre: Skipped 1 previous similar message [ 7668.747968] Lustre: Skipped 1 previous similar message [ 7673.829336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7673.842922] Lustre: Skipped 2 previous similar messages [ 7674.263157] Lustre: server umount lustre-MDT0000 complete [ 7678.953822] LustreError: 191729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7678.987121] LustreError: 191729:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7679.021844] LustreError: 191711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444629 with bad export cookie 14461248103524705906 [ 7679.041082] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7679.045240] LustreError: 191711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7679.401962] Lustre: server umount lustre-MDT0001 complete [ 7694.542840] Lustre: server umount lustre-OST0000 complete [ 7703.991831] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_hostid [ 7713.117766] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 7749.463348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7749.716115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7749.737184] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7749.806374] Lustre: lustre-MDT0000: new disk, initializing [ 7749.866298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7749.881861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7754.286129] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 7764.714491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7764.846866] Lustre: 197123:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7764.888628] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7764.893459] Lustre: Skipped 1 previous similar message [ 7764.895588] LustreError: 197903:0:(llog.c:1696:llog_backup()) MGC192.168.206.137@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7764.903973] LustreError: 197903:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7764.908078] Lustre: 197903:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.137@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7764.923649] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7764.929852] Lustre: Skipped 1 previous similar message [ 7764.982753] Lustre: lustre-MDT0001: new disk, initializing [ 7765.042303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7765.077583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7765.089139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7769.824280] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 7780.515810] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7786.522925] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7793.319639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7793.680124] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7793.682907] Lustre: Skipped 1 previous similar message [ 7793.687183] LustreError: 199065:0:(llog.c:1696:llog_backup()) MGC192.168.206.137@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7793.700774] LustreError: 199065:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7793.708434] Lustre: 199065:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.137@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7793.725408] Lustre: 199065:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7793.765324] Lustre: lustre-OST0000: new disk, initializing [ 7793.775599] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7793.783868] Lustre: 199090:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7793.906614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7800.452981] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 7802.961510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7802.971071] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7803.049795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7809.576302] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7813.854965] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7814.055825] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7818.309385] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7818.508410] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7823.849690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7823.855743] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7823.865445] Lustre: Skipped 1 previous similar message [ 7823.869190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7823.873409] Lustre: Skipped 1 previous similar message [ 7826.794369] Lustre: server umount lustre-OST0000 complete [ 7831.520366] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7831.529797] LustreError: Skipped 1 previous similar message [ 7831.535354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7831.545808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7834.105798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7836.560595] Lustre: server umount lustre-MDT0000 complete [ 7839.207890] LustreError: 197133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7839.246072] LustreError: 197133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7840.449297] LustreError: 199089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444791 with bad export cookie 14461248103524706627 [ 7840.463136] LustreError: 199089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7840.469050] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7840.846215] Lustre: server umount lustre-MDT0001 complete [ 7850.360659] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 10:13:19 (1788444799) [ 7869.081620] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 7878.649043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7878.991238] LustreError: 202216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7879.090357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7883.922739] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 7884.264075] LustreError: 202217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7884.293692] LustreError: 202217:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7892.369316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7897.159637] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 7899.915769] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7906.139787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7913.228793] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 7921.799368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7921.997542] Lustre: lustre-OST0001: new disk, initializing [ 7922.003240] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7922.015582] Lustre: 204524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7922.111064] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7922.120218] Lustre: Skipped 2 previous similar messages [ 7928.422169] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 7931.474123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7931.486512] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7931.616925] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7939.735603] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7946.473223] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7949.447869] Lustre: DEBUG MARKER: ADD OST3 [ 7954.516814] loop: module loaded [ 7954.568215] loop0: detected capacity change from 0 to 204800000 [ 7954.972597] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7955.005450] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7955.028372] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7962.843950] loop0: detected capacity change from 0 to 204800000 [ 7963.194092] Lustre: DEBUG MARKER: STOP OST3 [ 7969.975336] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7970.165920] Lustre: lustre-OST0003: new disk, initializing [ 7970.179579] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7970.188098] Lustre: 206572:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7972.117926] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7972.341701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7972.352677] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7972.389691] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7973.399845] Lustre: Failing over lustre-OST0003 [ 7973.528138] Lustre: server umount lustre-OST0003 complete [ 7974.178055] LustreError: 206729:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7976.452605] LustreError: 203698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7976.462890] LustreError: 203698:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7977.430693] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 7977.439852] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7977.443951] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7977.456904] Lustre: Skipped 1 previous similar message [ 7985.273622] Lustre: DEBUG MARKER: START OST3 again [ 7992.804795] LustreError: 203700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7992.820499] LustreError: 203700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 7992.820556] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7993.154945] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7993.157542] Lustre: Skipped 1 previous similar message [ 7993.169468] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7994.726425] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7994.830031] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7994.837023] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7999.547372] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 8019.597250] Lustre: Failing over lustre-OST0003 [ 8019.793406] Lustre: server umount lustre-OST0003 complete [ 8020.450915] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8020.451222] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8020.457358] LustreError: Skipped 1 previous similar message [ 8020.477638] Lustre: Skipped 1 previous similar message [ 8029.163665] LustreError: 207891:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8029.195403] LustreError: 207891:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 8036.320144] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8036.328648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8036.333525] Lustre: Skipped 2 previous similar messages [ 8039.402640] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8039.427749] Lustre: Skipped 3 previous similar messages [ 8042.878886] Lustre: server umount lustre-MDT0000 complete [ 8046.219544] LustreError: 204523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444996 with bad export cookie 14461248103524707509 [ 8046.223460] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8046.241513] LustreError: 204523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8049.657596] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8049.675656] Lustre: Skipped 5 previous similar messages [ 8052.877955] Lustre: server umount lustre-MDT0001 complete [ 8057.643656] Lustre: server umount lustre-OST0000 complete [ 8063.142399] Lustre: server umount lustre-OST0001 complete [ 8074.141896] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_hostid [ 8084.377927] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 8120.798762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8121.049543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8121.077412] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8121.154957] Lustre: lustre-MDT0000: new disk, initializing [ 8121.219416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8121.236415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8126.118611] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 8136.040295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8136.212700] Lustre: 211986:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8136.507990] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8142.360404] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 8154.096406] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8160.874373] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8168.111322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8168.461365] Lustre: lustre-OST0000: new disk, initializing [ 8168.463987] Lustre: Skipped 1 previous similar message [ 8168.469516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8168.476966] Lustre: Skipped 2 previous similar messages [ 8168.486825] Lustre: 213955:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8169.554728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8169.566585] Lustre: Skipped 1 previous similar message [ 8169.587420] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8169.692121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8175.047463] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 8186.907753] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8192.443213] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8192.799563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8198.226266] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8198.401399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8200.684288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8200.686588] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8200.695921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8200.695929] Lustre: Skipped 2 previous similar messages [ 8200.696809] Lustre: Skipped 3 previous similar messages [ 8200.728349] LustreError: Skipped 1 previous similar message [ 8206.873211] Lustre: server umount lustre-OST0000 complete [ 8216.771200] Lustre: server umount lustre-MDT0000 complete [ 8221.051884] LustreError: 211979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445171 with bad export cookie 14461248103524709798 [ 8221.058491] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8221.069206] LustreError: 211979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8221.863520] Lustre: server umount lustre-MDT0001 complete [ 8226.543363] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 10:19:35 (1788445175) [ 8244.100287] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_hostid [ 8253.187838] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 8297.735672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8298.196867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8298.267783] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8298.512351] Lustre: lustre-MDT0000: new disk, initializing [ 8298.801790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8306.649503] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 8316.925465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8317.064702] Lustre: 218570:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8317.455225] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8321.842887] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 8330.779974] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8336.161780] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8342.538624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8342.868857] Lustre: 220539:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8344.597642] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8348.758598] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 8357.354865] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8361.628184] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8361.825049] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8365.275134] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8365.454726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8370.147097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8370.159428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8370.160168] Lustre: Skipped 3 previous similar messages [ 8370.165744] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8370.166377] Lustre: Skipped 7 previous similar messages [ 8370.177869] LustreError: Skipped 1 previous similar message [ 8381.407800] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8381.573846] Lustre: server umount lustre-OST0000 complete [ 8390.619088] Lustre: server umount lustre-MDT0000 complete [ 8390.625471] LustreError: 218577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8390.636520] LustreError: 218577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 8393.681277] LustreError: 218562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445344 with bad export cookie 14461248103524710680 [ 8393.687899] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8393.689722] LustreError: 218562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8393.927711] Lustre: server umount lustre-MDT0001 complete [ 8401.806646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8402.325873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8402.330582] Lustre: Skipped 5 previous similar messages [ 8407.309825] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 8415.508101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8421.658296] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 8428.518992] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8433.633875] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8440.657288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8447.512408] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 8454.910902] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8465.889608] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8465.936769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8465.949874] Lustre: Skipped 8 previous similar messages [ 8467.777475] Lustre: server umount lustre-OST0000 complete [ 8479.917327] Lustre: server umount lustre-MDT0000 complete [ 8484.618366] LustreError: 222737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445435 with bad export cookie 14461248103524711569 [ 8484.642383] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8484.651751] LustreError: 222737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8485.619265] Lustre: server umount lustre-MDT0001 complete [ 8596.045602] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [ 8599.069866] Key type lgssc unregistered [ 8599.315919] LNet: 226422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8599.322675] LNetError: 226422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8599.336837] LNet: Removed LNI 192.168.206.137@tcp [ 8600.291178] Key type .llcrypt unregistered [ 8600.298374] Key type ._llcrypt unregistered [ 8628.579884] Key type ._llcrypt registered [ 8628.582744] Key type .llcrypt registered [ 8628.666988] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_hostid [ 8645.707945] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 8647.320892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8647.675425] alg: No test for adler32 (adler32-zlib) [ 8648.738906] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8649.100707] LNet: Added LNI 192.168.206.137@tcp [8/256/0/180] [ 8650.806113] Key type lgssc registered [ 8652.133193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8694.275972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8694.316674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8695.604512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8695.628111] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8695.686693] Lustre: lustre-MDT0000: new disk, initializing [ 8695.752828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8695.769558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8701.082918] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 8713.611597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8713.731682] Lustre: 229773:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8713.785460] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8713.789557] Lustre: Skipped 1 previous similar message [ 8713.860403] Lustre: lustre-MDT0001: new disk, initializing [ 8714.003862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8714.058352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8714.066811] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8718.797811] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 8729.988581] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8736.776525] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8744.118822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8744.421448] Lustre: lustre-OST0000: new disk, initializing [ 8744.426938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8744.442345] Lustre: 231738:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8744.524343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8750.729922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8750.745320] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8750.851517] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8751.580325] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 8762.306356] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8767.104446] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8767.400325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8771.769197] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8771.931979] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8776.674345] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8776.681256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8776.687949] LustreError: Skipped 1 previous similar message [ 8776.689137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8776.701417] Lustre: Skipped 1 previous similar message [ 8776.746710] Lustre: Skipped 1 previous similar message [ 8780.211125] Lustre: server umount lustre-OST0000 complete [ 8785.892573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8785.916489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8785.944831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8786.962337] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8786.978411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8786.983750] Lustre: Skipped 1 previous similar message [ 8792.037467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8792.043946] Lustre: Skipped 1 previous similar message [ 8797.152993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8797.157446] Lustre: Skipped 1 previous similar message [ 8798.175140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8798.510080] Lustre: server umount lustre-MDT0000 complete [ 8803.118965] LustreError: 233079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445753 with bad export cookie 13745012900874450994 [ 8803.123958] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8803.138265] LustreError: 233079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8803.578879] Lustre: server umount lustre-MDT0001 complete [ 8822.755898] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 8834.508714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8834.995975] LustreError: 234680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8835.027632] LustreError: 234680:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8835.091349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8840.165206] LustreError: 234681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8840.206114] LustreError: 234681:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8840.866718] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 8845.280899] LustreError: 234680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8850.337925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8850.400864] LustreError: 234681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8850.824137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8856.613820] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 8860.321653] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8866.587822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8866.889912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8872.248849] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 8880.117691] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8880.348837] Lustre: lustre-OST0001: new disk, initializing [ 8880.353501] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8880.359951] Lustre: 236988:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8885.854467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8885.871885] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8885.984271] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8886.484783] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 8895.772975] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8901.141396] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8913.195379] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 10:31:02 (1788445862) [ 8916.970898] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8916.980714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8917.001307] Lustre: Skipped 3 previous similar messages [ 8917.010055] Lustre: Skipped 3 previous similar messages [ 8921.765288] Lustre: server umount lustre-MDT0000 complete [ 8922.092818] LustreError: 235794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8922.110799] LustreError: 235794:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8925.304946] LustreError: 237755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445875 with bad export cookie 13745012900874451869 [ 8925.312647] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8925.318137] LustreError: 237755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8926.151129] Lustre: server umount lustre-MDT0001 complete [ 8940.076119] Lustre: server umount lustre-OST0000 complete [ 8943.071423] Lustre: 227995:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445877/real 1788445877] req@ffff98bec3d3e680 x1875321132524928/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788445893 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8943.099509] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8944.973561] Lustre: server umount lustre-OST0001 complete [ 8952.954797] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_hostid [ 8963.853986] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 8999.968371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9000.312418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9000.333033] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9000.414821] Lustre: lustre-MDT0000: new disk, initializing [ 9000.470551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9000.474282] Lustre: Skipped 1 previous similar message [ 9000.488403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9005.450955] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 9017.882614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9018.156886] Lustre: 240935:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9018.277326] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9018.287036] Lustre: Skipped 1 previous similar message [ 9018.382856] Lustre: lustre-MDT0001: new disk, initializing [ 9018.629707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9018.655320] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9024.571617] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 9034.292398] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9040.999494] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9048.440989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9048.723869] Lustre: lustre-OST0000: new disk, initializing [ 9048.727536] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9048.734651] Lustre: 242907:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9048.823934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9048.830207] Lustre: Skipped 1 previous similar message [ 9050.689179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9050.696366] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9050.806927] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9054.730734] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 9065.104693] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9070.375317] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9070.665675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9075.789814] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9076.009335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9081.828244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9081.829850] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9081.859188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9081.861976] LustreError: Skipped 1 previous similar message [ 9081.898476] Lustre: Skipped 1 previous similar message [ 9084.439556] Lustre: server umount lustre-OST0000 complete [ 9090.016747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9094.713969] Lustre: server umount lustre-MDT0000 complete [ 9097.188897] LustreError: 240943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9097.203316] LustreError: 240943:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9097.828104] LustreError: 240927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446048 with bad export cookie 13745012900874454011 [ 9097.831707] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9097.844828] LustreError: 240927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9098.161659] Lustre: server umount lustre-MDT0001 complete [ 9105.478312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9110.221883] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 9116.135063] LustreError: 245123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9116.157899] LustreError: 245123:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9119.396635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9119.891826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9119.903821] Lustre: Skipped 1 previous similar message [ 9124.213962] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 9132.084837] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9138.031180] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9144.084392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9150.427637] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 9156.787755] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9304.035156] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9304.049849] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9304.069414] Lustre: Skipped 3 previous similar messages [ 9304.080773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9304.089230] Lustre: Skipped 3 previous similar messages [ 9316.319804] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9316.677585] Lustre: server umount lustre-OST0000 complete [ 9324.517321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9334.239307] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9334.535145] Lustre: server umount lustre-MDT0000 complete [ 9335.272648] LustreError: 245124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9335.312645] LustreError: 245124:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9338.400078] LustreError: 245105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446289 with bad export cookie 13745012900874454886 [ 9338.409592] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9338.415935] LustreError: 245105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9338.795108] Lustre: server umount lustre-MDT0001 complete [ 9425.413161] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [ 9427.769292] Key type lgssc unregistered [ 9428.108877] LNet: 248734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9428.115379] LNetError: 248734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9428.145135] LNet: Removed LNI 192.168.206.137@tcp [ 9429.063157] Key type .llcrypt unregistered [ 9429.066348] Key type ._llcrypt unregistered [ 9455.405222] Key type ._llcrypt registered [ 9455.409381] Key type .llcrypt registered [ 9455.608416] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_hostid [ 9475.598296] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 9476.671034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9476.812390] alg: No test for adler32 (adler32-zlib) [ 9477.996394] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9478.235346] LNet: Added LNI 192.168.206.137@tcp [8/256/0/180] [ 9479.919176] Key type lgssc registered [ 9481.154369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9523.867677] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 9532.367265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9532.395217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9533.629721] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9533.650470] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9533.711252] Lustre: lustre-MDT0000: new disk, initializing [ 9533.772053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9533.790984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9537.859144] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 9547.322685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9547.416914] Lustre: 252801:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9547.463944] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9547.469347] Lustre: Skipped 1 previous similar message [ 9547.525378] Lustre: lustre-MDT0001: new disk, initializing [ 9547.639773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9547.676591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9547.689373] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9552.123847] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 9556.996520] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9564.671114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9565.050968] Lustre: lustre-OST0000: new disk, initializing [ 9565.055587] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9565.061988] Lustre: 254437:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9565.182991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9569.089762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9569.103400] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9569.208753] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9570.958196] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 9581.259224] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9581.478119] Lustre: lustre-OST0001: new disk, initializing [ 9581.492223] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9581.508393] Lustre: 255306:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9581.597697] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9587.033981] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 9589.825777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9589.838676] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9589.933715] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9597.416049] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9603.462857] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9614.320751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9614.337052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9614.353355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9615.852102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9615.853467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9615.855882] Lustre: Skipped 1 previous similar message [ 9615.866574] Lustre: Skipped 1 previous similar message [ 9620.689229] Lustre: server umount lustre-MDT0000 complete [ 9624.341225] LustreError: 252794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446574 with bad export cookie 1947367169950248595 [ 9624.356134] LustreError: 252794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9624.365952] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9624.690903] Lustre: server umount lustre-MDT0001 complete [ 9639.071260] Lustre: server umount lustre-OST0000 complete [ 9641.952285] Lustre: 250309:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788446576/real 1788446576] req@ffff98bfff816a00 x1875322001760640/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788446592 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9641.987225] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9642.015107] Lustre: Skipped 1 previous similar message [ 9643.019474] Lustre: server umount lustre-OST0001 complete [ 9656.642770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9665.557880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9672.973304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9681.586206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9697.476985] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 9706.524819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9706.562606] Lustre: lustre-MDT0000: reset Object Index mappings [ 9706.952660] LustreError: 260312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9706.966771] LustreError: 260312:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9707.042295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9711.176184] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 9712.098340] LustreError: 260313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9712.131215] LustreError: 260313:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9717.217928] LustreError: 260312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9720.012847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9720.032206] Lustre: lustre-MDT0001: reset Object Index mappings [ 9720.374094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9724.937844] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 9727.595743] Lustre: 261453:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9733.727513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9740.103802] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 9745.394389] LustreError: 261809:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9748.198718] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9748.365457] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9748.379333] Lustre: Skipped 1 previous similar message [ 9751.501584] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9751.504232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9754.680961] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 9762.942564] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9767.235179] Lustre: 263328:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9774.850062] Lustre: server umount lustre-MDT0000 complete [ 9776.611300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9776.622308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9776.638192] LustreError: 260313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9779.007225] LustreError: 263331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446729 with bad export cookie 1947367169950255385 [ 9779.028640] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9779.042690] LustreError: 263331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9779.419847] Lustre: server umount lustre-MDT0001 complete [ 9794.420829] Lustre: server umount lustre-OST0000 complete [ 9797.583332] Lustre: 250309:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788446732/real 1788446732] req@ffff98bfed50b480 x1875322001808768/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788446748 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9797.623297] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9797.637636] Lustre: Skipped 3 previous similar messages [ 9798.029102] Lustre: server umount lustre-OST0001 complete [ 9816.543943] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 10:46:05 (1788446765) [ 9830.753420] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_hostid [ 9839.507568] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 9872.484189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9872.677463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9872.697406] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9872.756228] Lustre: lustre-MDT0000: new disk, initializing [ 9872.815960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9872.835108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9878.104564] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 9889.943936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9890.114583] Lustre: 267584:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9890.189398] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9890.193996] Lustre: Skipped 1 previous similar message [ 9890.352270] Lustre: lustre-MDT0001: new disk, initializing [ 9890.636154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9890.649050] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9895.220657] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 9903.770523] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9909.330111] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9915.576851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9915.862942] Lustre: lustre-OST0000: new disk, initializing [ 9915.873626] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9915.884692] Lustre: 269553:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9917.130316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9917.138408] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9917.229320] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9921.814218] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [ 9930.204735] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9934.494933] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9934.724860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9938.779519] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9938.935611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9943.013030] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9943.021523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9943.028163] LustreError: Skipped 1 previous similar message [ 9943.035341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9943.044778] Lustre: Skipped 1 previous similar message [ 9943.063337] Lustre: Skipped 1 previous similar message [ 9946.801574] Lustre: server umount lustre-OST0000 complete [ 9952.226552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9952.235718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9956.470390] Lustre: server umount lustre-MDT0000 complete [ 9958.371197] LustreError: 267596:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9958.387372] LustreError: 267596:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9960.434261] LustreError: 268379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446911 with bad export cookie 1947367169950257478 [ 9960.446705] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9960.452040] LustreError: 268379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9960.789167] Lustre: server umount lustre-MDT0001 complete [ 9977.292456] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [ 9986.986664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9987.443551] LustreError: 272485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9987.518543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9987.530604] Lustre: Skipped 2 previous similar messages [ 9992.661476] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10002.308807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10007.404817] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10010.141654] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10017.379783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10023.813655] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10031.204596] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10031.350884] Lustre: lustre-OST0001: new disk, initializing [10031.354568] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10031.361269] Lustre: 274794:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10032.551534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10032.560154] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10032.689104] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10037.208551] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10046.294113] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10050.284022] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10055.951159] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10056.018392] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10067.840581] Lustre: server umount lustre-MDT0000 complete [10068.448135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10068.465144] LustreError: 273499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10068.469156] Lustre: Skipped 5 previous similar messages [10068.483894] LustreError: 273499:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10071.566251] LustreError: 272467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447022 with bad export cookie 1947367169950258360 [10071.569231] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10071.576089] LustreError: 272467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10071.911884] Lustre: server umount lustre-MDT0001 complete [10085.935261] Lustre: server umount lustre-OST0000 complete [10089.952177] Lustre: 250307:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788447024/real 1788447024] req@ffff98bec36e9500 x1875322001907968/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788447040 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10090.207419] Lustre: server umount lustre-OST0001 complete [10108.110044] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [10117.589902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10118.095063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10118.103452] Lustre: Skipped 3 previous similar messages [10122.395158] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10130.147487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10134.197454] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10136.801576] Lustre: 279278:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10142.657270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10148.302646] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10155.563862] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10161.377171] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10162.927502] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10162.936510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10168.838708] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10172.711926] Lustre: 281153:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10178.061136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10178.090174] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10190.381571] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 10:52:19 (1788447139) [10207.202662] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10207.213343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10207.221137] Lustre: Skipped 1 previous similar message [10207.231631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10207.236507] Lustre: Skipped 2 previous similar messages [10210.807963] Lustre: server umount lustre-MDT0000 complete [10213.824449] LustreError: 278117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447164 with bad export cookie 1947367169950264744 [10213.837466] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10214.390599] LustreError: 280413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10214.408623] LustreError: 280413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10218.400412] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10218.408916] Lustre: Skipped 7 previous similar messages [10228.193401] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10228.469201] Lustre: server umount lustre-MDT0001 complete [10238.178282] Lustre: server umount lustre-OST0000 complete [10241.835696] Lustre: server umount lustre-OST0001 complete [10249.781944] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_hostid [10259.249673] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [10303.207897] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [10312.033090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10312.260685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10312.284752] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10312.343282] Lustre: lustre-MDT0000: new disk, initializing [10312.486928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10316.091788] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10325.914677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10326.016518] Lustre: 285681:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10326.245577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10329.856693] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10339.201668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10339.418745] Lustre: 287313:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10341.241647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10344.462653] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10354.731618] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10354.882650] Lustre: 288182:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10356.852360] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10360.474126] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10370.264832] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10374.116424] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10374.121250] Lustre: Skipped 1 previous similar message [10388.919625] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 10:55:38 (1788447338) [10392.545491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10392.555124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10392.572250] Lustre: Skipped 8 previous similar messages [10392.589205] Lustre: Skipped 5 previous similar messages [10397.308579] Lustre: server umount lustre-MDT0000 complete [10397.674379] LustreError: 285686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10397.694846] LustreError: 285686:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10400.946699] LustreError: 285672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447351 with bad export cookie 1947367169950272675 [10400.956605] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10400.963504] LustreError: 285672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10401.378408] Lustre: server umount lustre-MDT0001 complete [10416.123388] Lustre: server umount lustre-OST0000 complete [10419.175943] Lustre: 250307:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788447353/real 1788447353] req@ffff98bfc84fce00 x1875322002053376/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788447369 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10420.157963] Lustre: server umount lustre-OST0001 complete [10428.387345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10428.794437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10428.803491] Lustre: Skipped 7 previous similar messages [10434.018259] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10443.207932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10449.175499] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10457.029219] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10462.557538] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10469.740892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10476.292964] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10484.435205] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10502.518657] loop1: detected capacity change from 0 to 204800000 [10502.764455] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10502.784037] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10502.803096] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10507.179899] loop1: detected capacity change from 0 to 204800000 [10511.585580] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10511.670261] LustreError: 290416:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs18594: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10511.680590] LustreError: 290416:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs18594 [10511.684562] LustreError: 290416:0:(mgs_handler.c:517:mgs_target_reg()) fs18594-MDT0001: Error checking target index: rc = -13 [10511.694655] LustreError: fs18594-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10511.702913] LustreError: 294146:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [10511.708406] LustreError: 294146:0:(tgt_mount.c:2132:server_put_super()) no obd fs18594-MDT0001 [10511.712551] LustreError: 294146:0:(tgt_mount.c:116:server_deregister_mount()) fs18594-MDT0001 not registered [10511.736553] Lustre: server umount fs18594-MDT0001 complete [10511.739696] LustreError: 294146:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10518.562368] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10518.696696] Lustre: Setting parameter fs18594-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs18594-MDT0001 [10518.751238] Lustre: srv-fs18594-MDT0001: No data found on store. Initialize space. [10518.756029] Lustre: Skipped 4 previous similar messages [10518.861907] Lustre: fs18594-MDT0001: new disk, initializing [10518.865568] Lustre: Skipped 3 previous similar messages [10519.024580] Lustre: srv-fs18594-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10519.048791] Lustre: 294612:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18594-MDT0001-mdtlov: can't create new llog: rc = -115 [10519.061321] LustreError: 294612:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18594-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10523.369992] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10529.180535] Lustre: server umount fs18594-MDT0001 complete [10541.592389] loop1: detected capacity change from 0 to 204800000 [10547.901035] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10548.298451] Lustre: 296181:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18594-MDT0001-mdtlov: can't create new llog: rc = -115 [10548.304783] LustreError: 296181:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18594-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10553.173100] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10553.317364] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10557.863605] Lustre: server umount fs18594-MDT0001 complete [10566.624278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10566.643521] Lustre: Skipped 1 previous similar message [10566.653224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10572.292560] Lustre: server umount lustre-MDT0000 complete [10580.809845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10580.952376] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10586.618849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10587.179933] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10595.132243] loop1: detected capacity change from 0 to 204800000 [10600.286209] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10600.603628] Lustre: 298628:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18594-MDT0001-mdtlov: can't create new llog: rc = -115 [10600.621325] LustreError: 298628:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18594-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10604.852555] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10609.380655] Lustre: server umount fs18594-MDT0001 complete [10632.162547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10632.189746] LustreError: Skipped 2 previous similar messages [10632.202952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10632.218552] Lustre: Skipped 4 previous similar messages [10637.519199] Lustre: server umount lustre-OST0000 complete [10647.279289] Lustre: server umount lustre-MDT0000 complete [10650.748420] LustreError: 297173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447601 with bad export cookie 1947367169950275356 [10650.762136] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10650.764348] LustreError: 297173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10651.130996] Lustre: server umount lustre-MDT0001 complete [10662.047088] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [10664.913684] Key type lgssc unregistered [10665.290487] LNet: 300969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10665.299622] LNetError: 300969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10665.323379] LNet: Removed LNI 192.168.206.137@tcp [10666.205135] Key type .llcrypt unregistered [10666.208257] Key type ._llcrypt unregistered [10684.579182] Key type ._llcrypt registered [10684.582807] Key type .llcrypt registered [10684.672224] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [10685.462801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10685.517475] alg: No test for adler32 (adler32-zlib) [10686.710503] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10686.954064] LNet: Added LNI 192.168.206.137@tcp [8/256/0/180] [10688.615936] Key type lgssc registered [10690.284529] Lustre: Echo OBD driver; http://www.lustre.org/ [10698.206160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10698.230480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10699.854786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10703.638521] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10709.996918] LustreError: 302241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10710.025455] LustreError: 302241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10712.035877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10712.347823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10716.299262] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10723.291419] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10728.268267] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10735.885165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10736.212804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10737.265811] LustreError: 304080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10742.464209] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10742.765297] LustreError: 304078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10744.807984] LustreError: 304080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10744.845710] LustreError: 304080:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10748.646616] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10749.948893] LustreError: 304081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10749.963928] LustreError: 304081:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10760.181874] LustreError: 304081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10760.201334] LustreError: 304081:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10763.607647] loop1: detected capacity change from 0 to 204800000 [10763.837549] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10763.861683] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10763.877432] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10768.441383] loop1: detected capacity change from 0 to 204800000 [10773.156274] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10773.309710] Lustre: fs3905-OST0000: new disk, initializing [10773.318174] Lustre: srv-fs3905-OST0000: No data found on store. Initialize space. [10773.329554] Lustre: 305983:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10773.415983] Lustre: fs3905-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10779.084622] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10780.661209] LustreError: 304078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10780.685358] LustreError: 304078:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [10786.665872] Lustre: server umount fs3905-OST0000 complete [10797.608554] loop1: detected capacity change from 0 to 204800000 [10802.629262] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10802.788531] Lustre: fs3905-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10808.684984] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10816.488060] LustreError: 304081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10816.512251] LustreError: 304081:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [10817.969481] loop2: detected capacity change from 0 to 204800000 [10818.180100] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10818.202040] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10818.213411] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10821.616876] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10824.501164] loop2: detected capacity change from 0 to 204800000 [10826.739032] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10831.033116] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10831.149299] LustreError: fs3905-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10831.166390] LustreError: fs3905-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10831.175068] LustreError: 309023:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [10831.181989] LustreError: 309023:0:(tgt_mount.c:2132:server_put_super()) no obd fs3905-OST0001 [10831.188449] LustreError: 309023:0:(tgt_mount.c:116:server_deregister_mount()) fs3905-OST0001 not registered [10831.233131] Lustre: server umount fs3905-OST0001 complete [10831.236499] LustreError: 309023:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10842.104816] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10842.299738] Lustre: fs3905-OST0001: new disk, initializing [10842.304693] Lustre: srv-fs3905-OST0001: No data found on store. Initialize space. [10842.325807] Lustre: 309575:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10842.445859] Lustre: fs3905-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10849.916549] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10861.303846] Lustre: server umount fs3905-OST0000 complete [10872.141721] Lustre: server umount fs3905-OST0001 complete [10883.047086] LustreError: 304159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10883.069821] LustreError: 304159:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [10884.351422] loop1: detected capacity change from 0 to 204800000 [10889.920937] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10890.236577] Lustre: fs3905-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10897.030785] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10903.888915] loop2: detected capacity change from 0 to 204800000 [10908.897260] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10915.045913] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [10923.581970] Lustre: server umount fs3905-OST0000 complete [10932.804874] Lustre: server umount fs3905-OST0001 complete [10947.560162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10947.563842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10947.585261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10947.594176] Lustre: Skipped 1 previous similar message [10947.605285] Lustre: Skipped 1 previous similar message [10952.672550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10952.689990] Lustre: Skipped 1 previous similar message [10957.793138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10957.817513] Lustre: Skipped 1 previous similar message [10960.356859] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10960.551623] Lustre: server umount lustre-OST0000 complete [10968.037730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10968.050475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10968.051513] Lustre: Skipped 1 previous similar message [10968.073374] Lustre: Skipped 1 previous similar message [10970.446630] Lustre: server umount lustre-MDT0000 complete [10974.116578] LustreError: 302221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447924 with bad export cookie 2692679855135751160 [10974.127777] LustreError: 302221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10974.130132] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10974.374746] Lustre: server umount lustre-MDT0001 complete [10985.658880] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [10988.299579] Key type lgssc unregistered [10988.647141] LNet: 315500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10988.659294] LNetError: 315500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10988.674185] LNet: Removed LNI 192.168.206.137@tcp [10989.589706] Key type .llcrypt unregistered [10989.594635] Key type ._llcrypt unregistered [11022.319164] Key type ._llcrypt registered [11022.320777] Key type .llcrypt registered [11022.407601] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_hostid [11038.234841] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [11039.304960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11039.492674] alg: No test for adler32 (adler32-zlib) [11040.579428] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11040.903145] LNet: Added LNI 192.168.206.137@tcp [8/256/0/180] [11042.552177] Key type lgssc registered [11043.426146] Lustre: Echo OBD driver; http://www.lustre.org/ [11080.581214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11080.604079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11081.775608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11081.796263] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11081.843985] Lustre: lustre-MDT0000: new disk, initializing [11081.944488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11081.973455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11087.026881] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [11099.622669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11099.851552] Lustre: 319356:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11099.909296] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11099.913852] Lustre: Skipped 1 previous similar message [11099.977968] Lustre: lustre-MDT0001: new disk, initializing [11100.064521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11100.099795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11100.123790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11105.533862] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [11114.473656] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11119.811387] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11126.499550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11126.711670] Lustre: lustre-OST0000: new disk, initializing [11126.715438] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11126.720854] Lustre: 321325:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11126.786353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11132.668929] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [11136.115568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11136.155673] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11136.319408] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11145.450453] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11148.363761] Lustre: Mounted lustre-client - version 2.17.58_2_gee80f9b [11150.075215] Lustre: Unmounted lustre-client [11162.080278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11162.082373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11162.097569] LustreError: Skipped 1 previous similar message [11162.099344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11162.110591] Lustre: Skipped 1 previous similar message [11162.132266] Lustre: Skipped 1 previous similar message [11163.618670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11163.627415] Lustre: Skipped 1 previous similar message [11168.207223] Lustre: server umount lustre-OST0000 complete [11173.857104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11173.862275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11173.876080] Lustre: Skipped 1 previous similar message [11178.347678] Lustre: server umount lustre-MDT0000 complete [11182.061025] LustreError: 319350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448132 with bad export cookie 17180675442681057722 [11182.065587] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11182.074781] LustreError: 319350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11182.417437] Lustre: server umount lustre-MDT0001 complete [11194.297583] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [11197.858238] Key type lgssc unregistered [11209.887276] LNet: 1 local NIs in recovery (showing 1): 192.168.206.137@tcp [11210.657944] LNet: 323488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11210.665391] LNetError: 323488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11278.250796] LNet: Removed LNI 192.168.206.137@tcp [11279.130421] Key type .llcrypt unregistered [11279.132538] Key type ._llcrypt unregistered [11295.864895] Key type ._llcrypt registered [11295.867364] Key type .llcrypt registered [11295.948293] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_hostid [11313.026721] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [11313.881606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11313.993412] alg: No test for adler32 (adler32-zlib) [11315.214247] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11315.416425] LNet: Added LNI 192.168.206.137@tcp [8/256/0/180] [11317.055187] Key type lgssc registered [11318.277536] Lustre: Echo OBD driver; http://www.lustre.org/ [11350.504527] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 11:11:39 (1788448299) [11357.936628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11357.992159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11359.247906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11359.289707] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11359.358066] Lustre: lustre-MDT0000: new disk, initializing [11359.434815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11359.458698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11364.057045] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [11375.041215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11375.174661] Lustre: 326710:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11375.260309] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11375.265039] Lustre: Skipped 1 previous similar message [11375.348726] Lustre: lustre-MDT0001: new disk, initializing [11375.505884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11375.556417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11375.568569] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11380.511854] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [11390.195194] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11396.466314] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11404.178215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11404.509602] Lustre: lustre-OST0000: new disk, initializing [11404.519538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11404.524450] Lustre: 328678:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11404.610880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11411.052323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11411.064560] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11411.207070] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11411.272878] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [11421.594218] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11437.024221] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11437.036614] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11437.037907] LustreError: Skipped 1 previous similar message [11437.038892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11437.067104] Lustre: Skipped 1 previous similar message [11442.152466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11442.177952] Lustre: Skipped 2 previous similar messages [11442.839337] Lustre: server umount lustre-OST0000 complete [11447.270065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11447.273858] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11447.288429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11447.296735] Lustre: Skipped 1 previous similar message [11447.314435] Lustre: Skipped 1 previous similar message [11452.385635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11452.399788] Lustre: Skipped 1 previous similar message [11453.563772] Lustre: server umount lustre-MDT0000 complete [11458.071273] LustreError: 326703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448408 with bad export cookie 17435751576788923747 [11458.075184] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11458.086469] LustreError: 326703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11458.460284] Lustre: server umount lustre-MDT0001 complete [11470.873492] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [11473.784576] Key type lgssc unregistered [11474.227515] LNet: 330586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11474.240470] LNetError: 330586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11474.264750] LNet: Removed LNI 192.168.206.137@tcp [11475.424722] Key type .llcrypt unregistered [11475.427842] Key type ._llcrypt unregistered [11499.505624] Key type ._llcrypt registered [11499.510067] Key type .llcrypt registered [11499.676843] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [11501.155866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11501.181379] alg: No test for adler32 (adler32-zlib) [11502.249698] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11502.497408] LNet: Added LNI 192.168.206.137@tcp [8/256/0/180] [11504.269551] Key type lgssc registered [11505.535123] Lustre: Echo OBD driver; http://www.lustre.org/ [11512.940570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11512.957395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11514.375538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11518.691159] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [11523.554018] LustreError: 331856:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11523.577608] LustreError: 331856:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11528.002708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11528.555351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11532.765522] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [11540.458323] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11546.022266] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11552.367503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11552.693406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11558.140839] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [11565.497077] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11574.708820] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11576.211431] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11579.872740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11579.879771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11579.892537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11582.451598] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.37@tcp (stopping) [11584.481448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11584.482636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11584.492291] Lustre: Skipped 2 previous similar messages [11585.751087] Lustre: server umount lustre-MDT0000 complete [11587.556403] LustreError: 331852:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11588.978403] LustreError: 331835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448539 with bad export cookie 8593573427290721805 [11588.988238] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11588.992091] LustreError: 331835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11589.601327] LustreError: 331853:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11589.605387] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11589.627796] Lustre: Skipped 1 previous similar message [11589.632659] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11592.664864] LustreError: 331852:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11593.637310] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11593.646772] Lustre: Skipped 1 previous similar message [11595.347879] Lustre: server umount lustre-MDT0001 complete [11599.570968] Lustre: server umount lustre-OST0000 complete [11608.797578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11613.155479] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [11615.932330] Lustre: 336037:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.137@tcp [11618.073357] Lustre: server umount MGS complete [11626.498781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11626.950522] LustreError: 336609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11626.965197] LustreError: 336609:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11627.057773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11631.365551] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [11637.218277] LustreError: 336610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11637.237764] LustreError: 336610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11640.838372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11641.238654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11646.034913] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [11655.623257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11655.916978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11662.857889] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [11672.378580] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11674.298797] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11681.306642] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11683.364672] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11692.003950] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11692.011268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11692.027658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11694.059299] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11703.263256] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11703.642130] Lustre: server umount lustre-OST0000 complete [11709.409303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11709.418659] Lustre: Skipped 1 previous similar message [11709.422734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11709.426773] Lustre: Skipped 4 previous similar messages [11722.210157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11722.420801] Lustre: server umount lustre-MDT0000 complete [11724.768598] LustreError: 336605:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11724.800642] LustreError: 336605:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11726.212172] LustreError: 339301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448676 with bad export cookie 8593573427290722645 [11726.214969] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11726.222197] LustreError: 339301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11726.490769] Lustre: server umount lustre-MDT0001 complete [11740.447443] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [11743.854137] Key type lgssc unregistered [11744.239247] LNet: 340054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11744.244645] LNetError: 340054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11744.273901] LNet: Removed LNI 192.168.206.137@tcp [11745.196345] Key type .llcrypt unregistered [11745.199046] Key type ._llcrypt unregistered [11766.560552] Key type ._llcrypt registered [11766.562514] Key type .llcrypt registered [11766.690592] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_hostid [11781.789273] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [11782.636230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11782.811327] alg: No test for adler32 (adler32-zlib) [11783.966731] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11784.326761] LNet: Added LNI 192.168.206.137@tcp [8/256/0/180] [11786.007227] Key type lgssc registered [11787.328778] Lustre: Echo OBD driver; http://www.lustre.org/ [11824.153399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11824.192171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11825.434473] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11825.516622] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11825.678692] Lustre: lustre-MDT0000: new disk, initializing [11825.769804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11825.797532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11830.830500] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [11841.472782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11841.585325] Lustre: 343208:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11841.637440] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11841.641036] Lustre: Skipped 1 previous similar message [11841.702697] Lustre: lustre-MDT0001: new disk, initializing [11841.792048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11841.830416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11841.840580] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11846.691880] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [11858.720499] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11865.376984] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11874.103299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11874.323287] Lustre: lustre-OST0000: new disk, initializing [11874.327865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11874.332598] Lustre: 345179:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11874.408541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11880.534172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11880.557226] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11880.701436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11881.573796] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [11891.531276] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11896.948096] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11897.188660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11902.119959] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11902.270211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11906.531077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11906.542393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11906.545173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11906.565787] Lustre: Skipped 1 previous similar message [11911.658468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11911.675932] Lustre: Skipped 2 previous similar messages [11916.769621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11916.775823] Lustre: Skipped 1 previous similar message [11918.303908] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11918.564266] Lustre: server umount lustre-OST0000 complete [11923.936588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11923.947682] LustreError: Skipped 1 previous similar message [11923.952740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11923.963600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11927.008920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11928.634956] Lustre: server umount lustre-MDT0000 complete [11931.940821] LustreError: 343201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448882 with bad export cookie 14021135967011415303 [11931.946488] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11931.949141] LustreError: 343201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11932.332702] Lustre: server umount lustre-MDT0001 complete [11941.332911] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 11:21:30 (1788448890) [11957.317353] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_hostid [11967.931928] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [12008.460422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12008.941130] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12008.990575] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12009.101259] Lustre: lustre-MDT0000: new disk, initializing [12009.265296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12009.285609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12013.828802] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [12023.963622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12024.158289] Lustre: 349945:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12024.212613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12024.220109] Lustre: Skipped 1 previous similar message [12024.291412] Lustre: lustre-MDT0001: new disk, initializing [12024.387647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12024.422518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12024.428917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12028.505544] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [12037.551280] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12042.920533] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12049.303439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12049.599215] Lustre: lustre-OST0000: new disk, initializing [12049.602395] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12049.609080] Lustre: 351911:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12049.682088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12055.188386] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [12056.642551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12056.652180] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12056.733083] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12064.213075] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12098.016042] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12098.017961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12098.026408] LustreError: Skipped 1 previous similar message [12098.028759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12098.028765] Lustre: Skipped 2 previous similar messages [12098.081618] Lustre: Skipped 1 previous similar message [12103.971249] Lustre: server umount lustre-OST0000 complete [12111.332277] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12111.342532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12111.364531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12111.373549] Lustre: Skipped 3 previous similar messages [12114.836949] Lustre: server umount lustre-MDT0000 complete [12118.893519] LustreError: 349937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449069 with bad export cookie 14021135967011416185 [12118.897595] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12118.906072] LustreError: 349937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12119.206935] Lustre: server umount lustre-MDT0001 complete [12131.868845] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [12134.660353] Key type lgssc unregistered [12135.038211] LNet: 354093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12135.051313] LNetError: 354093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12135.078665] LNet: Removed LNI 192.168.206.137@tcp [12135.989904] Key type .llcrypt unregistered [12135.996326] Key type ._llcrypt unregistered [12156.990960] Key type ._llcrypt registered [12156.993261] Key type .llcrypt registered [12157.098964] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [12158.183145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12158.320932] alg: No test for adler32 (adler32-zlib) [12159.454512] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12159.807402] LNet: Added LNI 192.168.206.137@tcp [8/256/0/180] [12161.439777] Key type lgssc registered [12162.692735] Lustre: Echo OBD driver; http://www.lustre.org/ [12171.155681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12171.189330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12172.796109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12177.886523] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [12183.016440] LustreError: 355363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12183.053064] LustreError: 355363:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12184.041592] LustreError: 355364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12189.159438] LustreError: 355363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12189.546517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12189.875329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12195.593901] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [12203.802211] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12209.922333] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12216.263903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12216.561318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12222.762157] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [12229.595878] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12245.478867] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12245.478961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12245.497387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12245.501899] LustreError: Skipped 1 previous similar message [12250.552352] Lustre: server umount lustre-OST0000 complete [12255.713380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12255.714297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12255.723674] Lustre: Skipped 2 previous similar messages [12255.737132] Lustre: Skipped 2 previous similar messages [12260.812714] Lustre: server umount lustre-MDT0000 complete [12260.839611] LustreError: 357206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12260.862280] LustreError: 357206:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12264.640179] LustreError: 355343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449215 with bad export cookie 3525407953555028067 [12264.649103] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12264.651528] LustreError: 355343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12265.261130] Lustre: server umount lustre-MDT0001 complete [12276.566697] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [12279.580354] Key type lgssc unregistered [12279.957082] LNet: 361005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12279.969612] LNetError: 361005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12279.993295] LNet: Removed LNI 192.168.206.137@tcp [12280.995152] Key type .llcrypt unregistered [12281.003702] Key type ._llcrypt unregistered [12298.552169] Key type ._llcrypt registered [12298.556582] Key type .llcrypt registered [12298.680164] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [12299.774968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12299.788530] alg: No test for adler32 (adler32-zlib) [12300.876368] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12301.168523] LNet: Added LNI 192.168.206.137@tcp [8/256/0/180] [12302.888280] Key type lgssc registered [12303.804903] Lustre: Echo OBD driver; http://www.lustre.org/ [12311.723505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12311.757737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12313.374611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12318.184856] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [12322.787309] LustreError: 365057:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12322.797353] LustreError: 365057:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12325.936121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12326.254828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12331.121535] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [12339.931320] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12348.275450] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12357.299901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12357.825650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12365.021733] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [12372.868949] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12381.670731] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12381.678701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12381.690791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12386.467513] Lustre: server umount lustre-OST0000 complete [12391.907626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12391.908828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12391.919158] Lustre: Skipped 2 previous similar messages [12391.932806] Lustre: Skipped 2 previous similar messages [12396.027742] Lustre: server umount lustre-MDT0000 complete [12397.033957] LustreError: 365051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12397.063880] LustreError: 365051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12399.283660] LustreError: 364977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449349 with bad export cookie 16879981553128211366 [12399.288946] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12399.290968] LustreError: 364977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12399.587721] Lustre: server umount lustre-MDT0001 complete [12408.805495] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [12411.660901] Key type lgssc unregistered [12412.016378] LNet: 372743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12412.035460] LNetError: 372743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12412.057275] LNet: Removed LNI 192.168.206.137@tcp [12413.064098] Key type .llcrypt unregistered [12413.065942] Key type ._llcrypt unregistered [12429.951717] Key type ._llcrypt registered [12429.954609] Key type .llcrypt registered [12430.106724] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [12431.082059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12431.138260] alg: No test for adler32 (adler32-zlib) [12432.253716] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12432.466348] LNet: Added LNI 192.168.206.137@tcp [8/256/0/180] [12434.257317] Key type lgssc registered [12435.801024] Lustre: Echo OBD driver; http://www.lustre.org/ [12446.054762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12446.103266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12447.711876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12453.978425] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [12457.975330] LustreError: 376753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12458.011972] LustreError: 376753:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12463.073646] LustreError: 376752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12465.384785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12465.821242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12470.794588] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [12480.747228] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12485.985144] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12494.034980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12494.414154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12500.944924] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [12508.503231] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12527.082558] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12527.089114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12527.100760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12528.097332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12528.114093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12528.121449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12528.135309] Lustre: Skipped 1 previous similar message [12532.914596] Lustre: server umount lustre-OST0000 complete [12542.947836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12542.967608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12542.998425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12546.337991] Lustre: server umount lustre-MDT0000 complete [12548.579437] LustreError: 376745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12548.608227] LustreError: 376745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12551.260761] LustreError: 376694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449501 with bad export cookie 12574294026207861554 [12551.274207] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12551.709242] Lustre: server umount lustre-MDT0001 complete [12563.006783] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [12565.591780] Key type lgssc unregistered [12566.021158] LNet: 384541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12566.026785] LNetError: 384541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12566.047466] LNet: Removed LNI 192.168.206.137@tcp [12567.031073] Key type .llcrypt unregistered [12567.034989] Key type ._llcrypt unregistered [12585.483188] Key type ._llcrypt registered [12585.491087] Key type .llcrypt registered [12585.554289] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [12586.256611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12586.329417] alg: No test for adler32 (adler32-zlib) [12587.412799] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12587.657690] LNet: Added LNI 192.168.206.137@tcp [8/256/0/180] [12589.343120] Key type lgssc registered [12590.706835] Lustre: Echo OBD driver; http://www.lustre.org/ [12598.460444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12598.503052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12600.044362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12604.725375] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [12609.506503] LustreError: 388648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12609.521624] LustreError: 388648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12614.631734] LustreError: 388649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12616.470463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12616.929210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12621.675960] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [12629.298476] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12635.840451] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12643.653029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12644.065801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12652.030883] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [12661.925154] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12677.093934] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12677.095232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12677.098708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12677.105076] LustreError: Skipped 1 previous similar message [12677.134761] Lustre: Skipped 1 previous similar message [12681.455185] Lustre: server umount lustre-OST0000 complete [12687.329184] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12687.358141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12687.366693] Lustre: Skipped 2 previous similar messages [12692.452662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12692.469057] Lustre: Skipped 1 previous similar message [12696.545342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12696.551268] Lustre: Skipped 1 previous similar message [12699.103122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12699.225277] Lustre: server umount lustre-MDT0000 complete [12701.665464] LustreError: 388643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12701.700497] LustreError: 388643:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12702.506593] LustreError: 388566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449653 with bad export cookie 7765964861842429729 [12702.522570] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12702.800794] Lustre: server umount lustre-MDT0001 complete [12712.730784] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [12715.029778] Key type lgssc unregistered [12715.288774] LNet: 394440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12715.292719] LNetError: 394440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12716.328634] LNet: Removed LNI 192.168.206.137@tcp [12717.111543] Key type .llcrypt unregistered [12717.114719] Key type ._llcrypt unregistered [12813.740368] Key type ._llcrypt registered [12813.743262] Key type .llcrypt registered [12813.832609] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [12814.132522] Key type .llcrypt unregistered [12814.135109] Key type ._llcrypt unregistered [12887.527787] Key type ._llcrypt registered [12887.532093] Key type .llcrypt registered [12887.639099] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [12887.869659] Key type .llcrypt unregistered [12887.876969] Key type ._llcrypt unregistered [12918.666229] Key type ._llcrypt registered [12918.670809] Key type .llcrypt registered [12918.797905] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [12919.163435] Key type .llcrypt unregistered [12919.167334] Key type ._llcrypt unregistered [12951.175530] Key type ._llcrypt registered [12951.177530] Key type .llcrypt registered [12951.258683] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [12951.482571] Key type .llcrypt unregistered [12951.489099] Key type ._llcrypt unregistered [13036.289537] Key type ._llcrypt registered [13036.293108] Key type .llcrypt registered [13036.405045] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [13036.734550] Key type .llcrypt unregistered [13036.736324] Key type ._llcrypt unregistered [13072.728218] Key type ._llcrypt registered [13072.730842] Key type .llcrypt registered [13072.835641] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [13073.131968] Key type .llcrypt unregistered [13073.135221] Key type ._llcrypt unregistered [13105.381407] Key type ._llcrypt registered [13105.384580] Key type .llcrypt registered [13105.520378] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [13105.807168] Key type .llcrypt unregistered [13105.809620] Key type ._llcrypt unregistered [13135.110662] Key type ._llcrypt registered [13135.113913] Key type .llcrypt registered [13135.207160] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [13135.496403] Key type .llcrypt unregistered [13135.499232] Key type ._llcrypt unregistered [13167.819885] Key type ._llcrypt registered [13167.826326] Key type .llcrypt registered [13167.972939] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [13168.295315] Key type .llcrypt unregistered [13168.301232] Key type ._llcrypt unregistered [13199.853901] Key type ._llcrypt registered [13199.856725] Key type .llcrypt registered [13200.021709] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [13200.304292] Key type .llcrypt unregistered [13200.306556] Key type ._llcrypt unregistered [13264.946931] Key type ._llcrypt registered [13264.948968] Key type .llcrypt registered [13265.060952] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [13265.305280] Key type .llcrypt unregistered [13265.307143] Key type ._llcrypt unregistered [13297.279773] Key type ._llcrypt registered [13297.283572] Key type .llcrypt registered [13297.425920] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [13297.734443] Key type .llcrypt unregistered [13297.737850] Key type ._llcrypt unregistered [13331.068226] Key type ._llcrypt registered [13331.071654] Key type .llcrypt registered [13331.128751] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [13331.378642] Key type .llcrypt unregistered [13331.381292] Key type ._llcrypt unregistered [13387.262784] Key type ._llcrypt registered [13387.264590] Key type .llcrypt registered [13387.364351] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_hostid [13401.033878] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [13401.814659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13401.846481] alg: No test for adler32 (adler32-zlib) [13402.873467] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13403.076126] LNet: Added LNI 192.168.206.137@tcp [8/256/0/180] [13404.719151] Key type lgssc registered [13405.630761] Lustre: Echo OBD driver; http://www.lustre.org/ [13439.141124] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13439.208163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13440.476797] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13440.509334] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13440.588265] Lustre: tst250fs-MDT0000: new disk, initializing [13440.667953] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13440.685747] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13445.674420] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [13454.600632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13454.709629] Lustre: 414968:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13454.754892] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13454.758669] Lustre: Skipped 1 previous similar message [13454.815878] Lustre: tst250fs-MDT0001: new disk, initializing [13454.902773] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13454.942671] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13454.954981] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13460.108568] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [13465.029277] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13471.969995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13472.148510] Lustre: tst250fs-OST0000: new disk, initializing [13472.151876] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13472.158056] Lustre: 416601:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13472.271767] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13478.477486] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13478.494242] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13478.597779] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [13478.627241] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13489.297210] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13489.531215] Lustre: tst250fs-OST0001: new disk, initializing [13489.537237] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13489.543943] Lustre: 417471:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13489.658484] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13496.423376] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_default_debug -1 all [13496.493642] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13496.504976] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13496.620968] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13501.891749] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13504.746472] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13504.755340] Lustre: Skipped 2 previous similar messages [13526.507369] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13526.518341] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13526.532203] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13527.522371] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13527.526690] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13527.544418] Lustre: Skipped 2 previous similar messages [13527.568420] Lustre: Skipped 3 previous similar messages [13529.708130] Lustre: server umount tst250fs-MDT0000 complete [13533.347115] LustreError: 417470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450483 with bad export cookie 9300321306828465399 [13533.361108] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13533.769536] Lustre: server umount tst250fs-MDT0001 complete [13548.117809] Lustre: server umount tst250fs-OST0000 complete [13562.344811] Lustre: server umount tst250fs-OST0001 complete [13581.641639] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_hostid [13591.352807] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [13627.175768] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 11:49:36 (1788450576) [13628.838694] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13644.333678] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing set_hostid [13655.598967] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing load_modules_local [13688.925994] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13410 sec ========= 11:50:37 (1788450637) [13690.895557] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:50:40 (1788450640) === [13694.756697] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:50:43 (1788450643) === [13737.162731] Lustre: DEBUG MARKER: oleg637-server.virtnet: executing unload_modules_local [13740.292717] Key type lgssc unregistered [13740.569530] LNet: 426615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13740.584498] LNetError: 426615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13741.606420] LNet: Removed LNI 192.168.206.137@tcp [13742.626178] Key type .llcrypt unregistered [13742.629594] Key type ._llcrypt unregistered