[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 478053891 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.003149] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005024] kvm-guest: setup PV IPIs [ 0.008684] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009020] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010041] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011026] pid_max: default: 32768 minimum: 301 [ 0.013192] LSM: Security Framework initializing [ 0.014093] Yama: becoming mindful. [ 0.016060] SELinux: Initializing. [ 0.017109] *** VALIDATE selinux *** [ 0.025865] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030701] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.032154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033140] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.035080] *** VALIDATE tmpfs *** [ 0.036566] *** VALIDATE proc *** [ 0.038211] *** VALIDATE cgroup *** [ 0.039012] *** VALIDATE cgroup2 *** [ 0.040361] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041212] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.044016] Spectre V2 : User space: Vulnerable [ 0.045013] Speculative Store Bypass: Vulnerable [ 0.048283] debug: unmapping init [mem 0xffffffff88859000-0xffffffff88860fff] [ 0.051170] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.052827] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.053031] ... version: 2 [ 0.054018] ... bit width: 48 [ 0.055018] ... generic registers: 4 [ 0.056017] ... value mask: 0000ffffffffffff [ 0.057019] ... max period: 00007fffffffffff [ 0.058018] ... fixed-purpose events: 3 [ 0.058920] ... event mask: 000000070000000f [ 0.059328] rcu: Hierarchical SRCU implementation. [ 0.061813] smp: Bringing up secondary CPUs ... [ 0.062654] x86: Booting SMP configuration: [ 0.063039] .... node #0, CPUs: #1 #2 #3 [ 0.067018] smp: Brought up 1 node, 4 CPUs [ 0.069027] smpboot: Max logical packages: 1 [ 0.070021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.232293] node 0 deferred pages initialised in 159ms [ 0.236641] devtmpfs: initialized [ 0.237280] x86/mm: Memory block size: 128MB [ 0.241102] gcov: version magic: 0x41383552 [ 0.244211] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.246117] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.247360] pinctrl core: initialized pinctrl subsystem [ 0.249135] [ 0.249628] ************************************************************* [ 0.252013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.254012] ** ** [ 0.256011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.257011] ** ** [ 0.259011] ** This means that this kernel is built to expose internal ** [ 0.260011] ** IOMMU data structures, which may compromise security on ** [ 0.263013] ** your system. ** [ 0.265014] ** ** [ 0.267013] ** If you see this message and you are not debugging the ** [ 0.269018] ** kernel, report this immediately to your vendor! ** [ 0.271014] ** ** [ 0.273014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.275013] ************************************************************* [ 0.277637] NET: Registered protocol family 16 [ 0.279382] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.280048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.282055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.284523] cpuidle: using governor menu [ 0.285496] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.288539] PCI: Using configuration type 1 for base access [ 0.290137] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.300209] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.303121] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.307040] cryptd: max_cpu_qlen set to 1000 [ 0.309246] ACPI: Added _OSI(Module Device) [ 0.311015] ACPI: Added _OSI(Processor Device) [ 0.312014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.314014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.318287] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.325170] ACPI: Interpreter enabled [ 0.326053] ACPI: PM: (supports S0 S3 S4 S5) [ 0.328030] ACPI: Using IOAPIC for interrupt routing [ 0.329130] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.332457] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.344535] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.346049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.349021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.352086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.357264] acpiphp: Slot [2] registered [ 0.358108] acpiphp: Slot [5] registered [ 0.359092] acpiphp: Slot [6] registered [ 0.360089] acpiphp: Slot [7] registered [ 0.361072] acpiphp: Slot [8] registered [ 0.362071] acpiphp: Slot [9] registered [ 0.363064] acpiphp: Slot [10] registered [ 0.364068] acpiphp: Slot [3] registered [ 0.365079] acpiphp: Slot [4] registered [ 0.366063] acpiphp: Slot [11] registered [ 0.367129] acpiphp: Slot [12] registered [ 0.368059] acpiphp: Slot [13] registered [ 0.369114] acpiphp: Slot [14] registered [ 0.371120] acpiphp: Slot [15] registered [ 0.372099] acpiphp: Slot [16] registered [ 0.374127] acpiphp: Slot [17] registered [ 0.375129] acpiphp: Slot [18] registered [ 0.377111] acpiphp: Slot [19] registered [ 0.378103] acpiphp: Slot [20] registered [ 0.379115] acpiphp: Slot [21] registered [ 0.381105] acpiphp: Slot [22] registered [ 0.382104] acpiphp: Slot [23] registered [ 0.384094] acpiphp: Slot [24] registered [ 0.385156] acpiphp: Slot [25] registered [ 0.387094] acpiphp: Slot [26] registered [ 0.388101] acpiphp: Slot [27] registered [ 0.390117] acpiphp: Slot [28] registered [ 0.391091] acpiphp: Slot [29] registered [ 0.392086] acpiphp: Slot [30] registered [ 0.394087] acpiphp: Slot [31] registered [ 0.395042] PCI host bridge to bus 0000:00 [ 0.396025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.399022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.402024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.404024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.407023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.409023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.411115] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.415069] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.416911] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.424013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.428073] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.430015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.431017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.432012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.434462] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.435553] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.437027] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.439069] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.442761] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.452032] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.457017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.463668] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.477017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.487021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.502019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.513775] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.523021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.546027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.592019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.603549] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.613033] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.622021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.641027] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.652000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.665019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.672020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.689023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.699189] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.714017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.728025] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.745019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.758816] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.769021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.778017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.799032] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.808400] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.810301] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.812358] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.814334] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.816211] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.820143] iommu: Default domain type: Passthrough [ 0.821387] SCSI subsystem initialized [ 0.823112] ACPI: bus type USB registered [ 0.824127] usbcore: registered new interface driver usbfs [ 0.825053] usbcore: registered new interface driver hub [ 0.827063] usbcore: registered new device driver usb [ 0.828119] pps_core: LinuxPPS API ver. 1 registered [ 0.829010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.831058] PTP clock support registered [ 0.833127] EDAC MC: Ver: 3.0.0 [ 0.834353] PCI: Using ACPI for IRQ routing [ 0.836706] NetLabel: Initializing [ 0.837010] NetLabel: domain hash size = 128 [ 0.838011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.840064] NetLabel: unlabeled traffic allowed by default [ 0.842162] vgaarb: loaded [ 0.843261] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.845014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.851284] clocksource: Switched to clocksource kvm-clock [ 0.956877] VFS: Disk quotas dquot_6.6.0 [ 0.958222] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.960105] *** VALIDATE ramfs *** [ 0.961279] *** VALIDATE hugetlbfs *** [ 0.963448] pnp: PnP ACPI init [ 0.965744] pnp: PnP ACPI: found 6 devices [ 0.980445] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.982903] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.984545] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.986027] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.987854] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.989724] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.991813] NET: Registered protocol family 2 [ 0.993781] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.998104] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.000996] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.005624] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.008359] TCP: Hash tables configured (established 65536 bind 65536) [ 1.010692] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.012861] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.015218] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.017675] NET: Registered protocol family 1 [ 1.020567] RPC: Registered named UNIX socket transport module. [ 1.022484] RPC: Registered udp transport module. [ 1.023693] RPC: Registered tcp transport module. [ 1.026210] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.029206] NET: Registered protocol family 44 [ 1.031169] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.033621] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.035881] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.038266] PCI: CLS 0 bytes, default 64 [ 1.040067] Unpacking initramfs... [ 2.442647] debug: unmapping init [mem 0xffff9f447cc54000-0xffff9f447ffbffff] [ 2.446599] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.449107] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.452149] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.000511] Initialise system trusted keyrings [ 3.002203] Key type blacklist registered [ 3.004037] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.011075] zbud: loaded [ 3.014630] *** VALIDATE nfs *** [ 3.016151] *** VALIDATE nfs4 *** [ 3.017743] pstore: using deflate compression [ 3.021939] Platform Keyring initialized [ 3.129475] NET: Registered protocol family 38 [ 3.130725] Key type asymmetric registered [ 3.131716] Asymmetric key parser 'x509' registered [ 3.133119] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.135119] io scheduler mq-deadline registered [ 3.136211] io scheduler kyber registered [ 3.137358] io scheduler bfq registered [ 3.138638] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.140661] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.142845] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.144860] ACPI: Power Button [PWRF] [ 3.151131] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.156582] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.167949] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.173941] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.184752] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.209607] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.237680] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.241768] Non-volatile memory driver v1.3 [ 3.243571] Linux agpgart interface v0.103 [ 3.272523] virtio_blk virtio1: [vda] 136552 512-byte logical blocks (69.9 MB/66.7 MiB) [ 3.274506] vda: detected capacity change from 0 to 69914624 [ 3.286980] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.289409] vdb: detected capacity change from 0 to 1073741824 [ 3.301219] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.303081] vdc: detected capacity change from 0 to 2621440000 [ 3.317729] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.319833] vdd: detected capacity change from 0 to 2621440000 [ 3.332136] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.334039] vde: detected capacity change from 0 to 4294967296 [ 3.346786] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.349244] vdf: detected capacity change from 0 to 4294967296 [ 3.354435] libphy: Fixed MDIO Bus: probed [ 3.363450] usbcore: registered new interface driver usbserial_generic [ 3.365537] usbserial: USB Serial support registered for generic [ 3.367588] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.371446] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.372897] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.375481] mousedev: PS/2 mouse device common for all mice [ 3.379197] rtc_cmos 00:05: RTC can wake from S4 [ 3.381602] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.384503] rtc_cmos 00:05: registered as rtc0 [ 3.390306] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.391699] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.396792] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.397203] intel_pstate: CPU model not supported [ 3.403069] hid: raw HID events driver (C) Jiri Kosina [ 3.405296] usbcore: registered new interface driver usbhid [ 3.407588] usbhid: USB HID core driver [ 3.409386] drop_monitor: Initializing network drop monitor service [ 3.412182] Initializing XFRM netlink socket [ 3.414358] NET: Registered protocol family 10 [ 3.417830] Segment Routing with IPv6 [ 3.419455] NET: Registered protocol family 17 [ 3.421619] mpls_gso: MPLS GSO support [ 3.427306] RAS: Correctable Errors collector initialized. [ 3.429665] AVX version of gcm_enc/dec engaged. [ 3.430812] AES CTR mode by8 optimization enabled [ 3.508389] sched_clock: Marking stable (3508310471, 0)->(4487193715, -978883244) [ 3.512189] registered taskstats version 1 [ 3.514258] Loading compiled-in X.509 certificates [ 3.516597] zswap: loaded using pool lzo/zbud [ 3.537075] Key type big_key registered [ 3.550226] Key type encrypted registered [ 3.551413] ima: No TPM chip found, activating TPM-bypass! [ 3.552755] ima: Allocated hash algorithm: sha1 [ 3.553963] ima: No architecture policies found [ 3.555427] evm: Initialising EVM extended attributes: [ 3.556800] evm: security.selinux [ 3.557674] evm: security.ima [ 3.558377] evm: security.capability [ 3.559283] evm: HMAC attrs: 0x1 [ 3.561238] rtc_cmos 00:05: setting system clock to 2026-06-12 04:28:08 UTC (1781238488) [ 3.565644] debug: unmapping init [mem 0xffffffff89803000-0xffffffff899fffff] [ 3.567858] debug: unmapping init [mem 0xffffffff88582000-0xffffffff88858fff] [ 3.575117] Write protecting the kernel read-only data: 28672k [ 3.577664] debug: unmapping init [mem 0xffffffff86c03000-0xffffffff86dfffff] [ 3.580989] debug: unmapping init [mem 0xffffffff87514000-0xffffffff875fffff] [ 3.614417] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.621582] systemd[1]: Detected virtualization kvm. [ 3.623272] systemd[1]: Detected architecture x86-64. [ 3.625190] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.652200] systemd[1]: No hostname configured. [ 3.653816] systemd[1]: Set hostname to . [ 3.655958] random: systemd: uninitialized urandom read (16 bytes read) [ 3.658318] systemd[1]: Initializing machine ID from random generator. [ 3.788151] random: systemd: uninitialized urandom read (16 bytes read) [ 3.790715] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.794438] random: systemd: uninitialized urandom read (16 bytes read) [ 3.796383] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.801345] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.381075] device-mapper: uevent: version 1.0.3 [ 4.384141] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.039525] random: fast init done [ 5.041564] virtio_net virtio0 ens2: renamed from eth0 [ 5.127515] scsi host0: ata_piix [ 5.137873] scsi host1: ata_piix [ 5.143152] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.145864] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.849027] dracut-initqueue[577]: RTNETLINK answers: File exists [ 10.073903] random: crng init done [ 10.075460] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.385390] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.331778] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.543308] SELinux: Disabled at runtime. [ 11.600732] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.608905] systemd[1]: Detected virtualization kvm. [ 11.610716] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.035970] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.038992] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.043302] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.048339] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.051652] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.058110] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.064909] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 12.129567] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.438965] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.747593] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.800581] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.866834] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.878592] EDAC sbridge: Ver: 1.1.2 [ 14.167670] Key type dns_resolver registered [ 14.459646] NFS: Registering the id_resolver key type [ 14.461474] Key type id_resolver registered [ 14.462855] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg110-server login: [ 69.483441] libcfs: loading out-of-tree module taints kernel. [ 69.522334] Key type ._llcrypt registered [ 69.524527] Key type .llcrypt registered [ 69.625440] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 89.583932] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 91.673644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 91.687495] alg: No test for adler32 (adler32-zlib) [ 93.035333] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 94.173292] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 96.072468] Key type lgssc registered [ 98.202578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 115.775126] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 158.601985] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 172.448536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 172.494879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 173.897210] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 173.997313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 174.208613] Lustre: lustre-MDT0000: new disk, initializing [ 174.376703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 174.405884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 179.902921] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.986818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 194.136875] Lustre: 6507:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 194.178554] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 194.186411] Lustre: Skipped 1 previous similar message [ 194.306986] Lustre: lustre-MDT0001: new disk, initializing [ 194.389231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 194.427907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 194.462901] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 198.675615] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.537018] hrtimer: interrupt took 12451955 ns [ 203.538540] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 214.743808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 215.323506] Lustre: lustre-OST0000: new disk, initializing [ 215.328584] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 215.338858] Lustre: 8412:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 215.469352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 215.532067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 215.579599] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 215.712051] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 221.824680] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.349173] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 238.497833] Lustre: lustre-OST0001: new disk, initializing [ 238.500861] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 238.506326] Lustre: 9468:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 238.596813] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 244.269432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 244.274932] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 244.381653] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 246.594692] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 261.574745] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 268.509386] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 276.075145] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing check_logdir /tmp/testlogs/ [ 281.679684] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing yml_node [ 286.600097] Lustre: DEBUG MARKER: Client: 2.17.53.70 [ 289.642793] Lustre: DEBUG MARKER: MDS: 2.17.53.70 [ 292.825689] Lustre: DEBUG MARKER: OSS: 2.17.53.70 [ 294.592853] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jun 12 00:32:57 EDT 2026 [ 313.777055] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 315.801712] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 332.770967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 332.779765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 332.794165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 336.352941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 336.352949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 336.368130] Lustre: Skipped 1 previous similar message [ 338.492138] Lustre: server umount lustre-MDT0000 complete [ 341.893720] LustreError: 7439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238826 with bad export cookie 4992136700066629805 [ 341.905621] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 341.921550] LustreError: 7439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 346.601457] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 346.608485] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 346.610889] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 346.610898] Lustre: Skipped 3 previous similar messages [ 346.633635] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 346.703086] Lustre: Skipped 2 previous similar messages [ 351.714030] LustreError: 9918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 351.715947] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 351.751165] LustreError: 9918:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 351.757035] Lustre: Skipped 2 previous similar messages [ 352.738771] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 352.749921] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 356.320291] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 356.709129] Lustre: server umount lustre-MDT0001 complete [ 361.318507] Lustre: server umount lustre-OST0000 complete [ 365.780736] Lustre: server umount lustre-OST0001 complete [ 375.456474] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 386.331110] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 428.302742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 428.751686] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 428.826297] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 428.948691] Lustre: lustre-MDT0000: new disk, initializing [ 429.059245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 429.090700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 433.785860] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 444.810186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 445.003291] Lustre: 14990:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 445.059931] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 445.063290] Lustre: Skipped 1 previous similar message [ 445.127703] Lustre: lustre-MDT0001: new disk, initializing [ 445.222645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 445.255827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 445.269474] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 449.142248] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 456.146110] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 459.929789] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 466.500263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 466.947926] Lustre: lustre-OST0000: new disk, initializing [ 466.954128] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 466.959872] Lustre: 16729:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 467.081569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 468.099629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 468.112422] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 468.257366] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 473.627420] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 481.573134] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 486.796149] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 487.009538] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 491.882529] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 492.104225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 494.053046] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 494.057062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 494.065983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 494.065991] Lustre: Skipped 2 previous similar messages [ 494.082880] Lustre: Skipped 1 previous similar message [ 504.307660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 504.317420] Lustre: Skipped 4 previous similar messages [ 508.384284] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 508.655512] Lustre: server umount lustre-OST0000 complete [ 514.533084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 519.024660] Lustre: server umount lustre-MDT0000 complete [ 519.653547] LustreError: 14994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 519.664610] LustreError: 14994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 522.844942] LustreError: 14982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239007 with bad export cookie 4992136700066631338 [ 522.863579] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 523.409549] Lustre: server umount lustre-MDT0001 complete [ 529.448737] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 00:36:52 (1781239012) [ 549.796103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 550.169269] LustreError: 19457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.243900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 554.877433] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 560.611386] LustreError: 19457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 560.634563] LustreError: 19457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 564.627585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.701367] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 576.222624] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 580.531924] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 582.680585] Lustre: Failing over lustre-MDT0000 [ 582.979289] Lustre: server umount lustre-MDT0000 complete [ 585.700725] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 585.702061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 585.704934] LustreError: Skipped 1 previous similar message [ 585.707364] LustreError: 19458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 585.721145] Lustre: Skipped 2 previous similar messages [ 585.768555] LustreError: 19458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 602.080087] Lustre: 3660:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781239070/real 1781239070] req@ffff9f44e195b0c0 x1867764024379136/t0(0) o400->MGC192.168.201.110@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781239086 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 602.139704] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 602.596624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 612.672952] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 617.039733] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 617.963664] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 617.982543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 618.016499] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 624.952826] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 632.293970] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 632.299890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 632.309809] Lustre: Skipped 1 previous similar message [ 632.325914] Lustre: Skipped 2 previous similar messages [ 635.675954] Lustre: server umount lustre-MDT0000 complete [ 639.461135] LustreError: 19458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 639.482533] LustreError: 19458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 640.074493] LustreError: 19439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239125 with bad export cookie 4992136700066632752 [ 640.101660] LustreError: 19439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 640.126316] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 640.652416] Lustre: server umount lustre-MDT0001 complete [ 649.847078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 650.585482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 650.597934] Lustre: Skipped 2 previous similar messages [ 654.945826] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 663.644086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 669.498391] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 675.453729] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 679.186637] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 681.143956] Lustre: Failing over lustre-MDT0000 [ 681.541926] Lustre: server umount lustre-MDT0000 complete [ 684.513408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 684.520560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 700.449587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 700.589405] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 700.881677] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 704.813673] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 706.023810] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 706.028971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 706.043115] Lustre: Skipped 1 previous similar message [ 706.061044] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 712.211392] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 716.258300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 716.267465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 716.270111] Lustre: Skipped 2 previous similar messages [ 722.663336] Lustre: server umount lustre-MDT0000 complete [ 725.475816] LustreError: 22450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 725.503483] LustreError: 22450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 726.512790] LustreError: 23173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239211 with bad export cookie 4992136700066633690 [ 726.520782] LustreError: 23173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 726.524344] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 726.824019] Lustre: server umount lustre-MDT0001 complete [ 735.034461] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 00:40:18 (1781239218) [ 752.854573] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 762.332851] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 803.971598] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 815.655150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 815.963660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 816.003414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 816.096888] Lustre: lustre-MDT0000: new disk, initializing [ 816.199534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 816.207787] Lustre: Skipped 2 previous similar messages [ 816.239504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 821.451061] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 832.698056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 832.815773] Lustre: 28439:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 833.116549] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 837.643066] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 846.220748] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 850.205344] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 857.506561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 857.805497] Lustre: 30179:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 859.462518] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 859.504557] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 859.569526] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 859.598545] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 864.193714] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 872.016597] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 878.095376] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 900.579872] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 900.580134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 900.595393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 900.614164] Lustre: Skipped 4 previous similar messages [ 900.620721] Lustre: Skipped 3 previous similar messages [ 902.864288] Lustre: server umount lustre-OST0000 complete [ 913.233565] Lustre: server umount lustre-MDT0000 complete [ 915.937670] LustreError: 28449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 917.106486] LustreError: 28431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239402 with bad export cookie 4992136700066634096 [ 917.115829] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 917.118925] LustreError: 28431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 917.743912] Lustre: server umount lustre-MDT0001 complete [ 929.384644] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 932.360299] Key type lgssc unregistered [ 932.673165] LNet: 32029:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.677088] LNetError: 32029:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.688084] LNet: Removed LNI 192.168.201.110@tcp [ 933.762131] Key type .llcrypt unregistered [ 933.765208] Key type ._llcrypt unregistered [ 959.528763] Key type ._llcrypt registered [ 959.531701] Key type .llcrypt registered [ 959.639965] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 960.337303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 960.416368] alg: No test for adler32 (adler32-zlib) [ 961.595535] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 961.909349] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 963.659177] Key type lgssc registered [ 965.221870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 973.182054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 973.214566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 974.776391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 978.798300] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 985.063388] LustreError: 33429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 985.088392] LustreError: 33429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 986.528704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 986.829573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 991.686329] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 997.776973] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1001.370186] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1008.841718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1009.269630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1011.360420] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1016.435809] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1019.535247] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1023.280239] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1042.494863] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 00:45:24 (1781239524) [ 1045.098843] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1047.267974] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1055.195598] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 00:45:38 (1781239538) [ 1065.922349] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 00:45:48 (1781239548) [ 1067.061180] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1079.546039] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1079.552080] Lustre: Skipped 2 previous similar messages [ 1082.001308] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 00:46:04 (1781239564) [ 1084.598402] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1084.610200] Lustre: Skipped 2 previous similar messages [ 1093.218397] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1093.222370] Lustre: Skipped 4 previous similar messages [ 1110.731826] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 00:46:33 (1781239593) [ 1114.595695] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1118.885315] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1129.015447] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 00:46:52 (1781239612) [ 1131.185377] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1131.191326] Lustre: Skipped 6 previous similar messages [ 1146.162761] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 00:47:09 (1781239629) [ 1160.167370] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1160.178762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1160.199847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1163.245732] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1163.250763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1163.259488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1163.280777] Lustre: Skipped 1 previous similar message [ 1165.262946] Lustre: server umount lustre-OST0000 complete [ 1171.441079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1171.456583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1171.487435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1173.475451] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1173.497293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1173.499768] Lustre: Skipped 1 previous similar message [ 1176.565902] Lustre: server umount lustre-MDT0000 complete [ 1178.604989] LustreError: 33430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1178.656753] LustreError: 33430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1182.038900] LustreError: 33410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239666 with bad export cookie 8941158792827944452 [ 1182.063625] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1182.068980] LustreError: 33410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1182.728700] Lustre: server umount lustre-MDT0001 complete [ 1197.659663] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 1201.582748] Key type lgssc unregistered [ 1202.016864] LNet: 40936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1202.020519] LNetError: 40936:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1203.065801] LNet: Removed LNI 192.168.201.110@tcp [ 1204.242795] Key type .llcrypt unregistered [ 1204.245291] Key type ._llcrypt unregistered [ 1228.453749] Key type ._llcrypt registered [ 1228.455543] Key type .llcrypt registered [ 1228.585098] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 1245.783539] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 1247.070584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1247.233819] alg: No test for adler32 (adler32-zlib) [ 1248.380294] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 1248.674879] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 1250.408574] Key type lgssc registered [ 1251.955324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1293.945750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1293.958944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1295.271344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1295.301703] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1295.365354] Lustre: lustre-MDT0000: new disk, initializing [ 1295.478254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1295.498939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1301.383179] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1312.332640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1312.441267] Lustre: 44058:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1312.469851] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1312.472525] Lustre: Skipped 1 previous similar message [ 1312.519611] Lustre: lustre-MDT0001: new disk, initializing [ 1312.605090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1312.643081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1312.653912] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1317.205561] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1325.494600] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1329.574536] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1335.924044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1336.213765] Lustre: lustre-OST0000: new disk, initializing [ 1336.218307] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1336.228438] Lustre: 45801:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1336.343032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1342.437798] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1345.608441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1345.614728] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1351.009305] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1351.180623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1356.098573] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1356.419505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1361.574121] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1361.828806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1366.505198] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1366.509495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1366.518690] Lustre: Skipped 1 previous similar message [ 1366.521438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1366.527988] Lustre: Skipped 1 previous similar message [ 1369.889855] Lustre: server umount lustre-OST0000 complete [ 1374.189160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1374.198462] LustreError: Skipped 1 previous similar message [ 1374.203826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1374.222686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1376.738816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1376.738870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1376.764459] Lustre: Skipped 1 previous similar message [ 1382.651626] Lustre: server umount lustre-MDT0000 complete [ 1386.814762] LustreError: 44049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239871 with bad export cookie 4923549631510178772 [ 1386.818229] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1386.821915] LustreError: 44049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1387.341785] Lustre: server umount lustre-MDT0001 complete [ 1398.813953] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 00:51:21 (1781239881) [ 1407.316281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1407.960868] LustreError: 48116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1407.985441] LustreError: 48116:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1408.085939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1412.809039] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1413.090962] LustreError: 48116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1413.126148] LustreError: 48116:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1417.209259] LustreError: 48117:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1422.309296] LustreError: 48116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1424.635073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1425.344132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1431.012578] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1438.508741] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1444.458765] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1454.250571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1454.799121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1463.182367] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1469.666141] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1484.641942] LustreError: 50331:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1485.855895] Lustre: Modifying parameter general.timeout=1 in log params [ 1487.662047] Lustre: Modifying parameter general.timeout=3 in log params [ 1487.666750] Lustre: Skipped 1 previous similar message [ 1489.888954] Lustre: Modifying parameter general.timeout=5 in log params [ 1489.896316] Lustre: Skipped 1 previous similar message [ 1494.508102] Lustre: Modifying parameter general.timeout=9 in log params [ 1494.512394] Lustre: Skipped 3 previous similar messages [ 1503.090771] Lustre: Modifying parameter general.timeout=18 in log params [ 1503.094321] Lustre: Skipped 8 previous similar messages [ 1519.472190] Lustre: Modifying parameter general.timeout=35 in log params [ 1519.476245] Lustre: Skipped 16 previous similar messages [ 1551.940362] Lustre: Modifying parameter general.timeout=69 in log params [ 1551.947801] Lustre: Skipped 33 previous similar messages [ 1625.975813] Lustre: Disabling parameter general.timeout= in log params [ 1625.980409] Lustre: Skipped 60 previous similar messages [ 1629.082397] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 00:55:12 (1781240112) [ 1631.926854] Lustre: 57277:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1634.128285] Lustre: 57385:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1637.864807] Lustre: 57581:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1637.896165] Lustre: 57581:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1642.603748] Lustre: 57792:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1642.614280] Lustre: 57792:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1669.299848] Lustre: 59155:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1669.329937] LustreError: 59155:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1673.211333] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 00:55:56 (1781240156) [ 1684.963710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1684.972121] Lustre: Skipped 2 previous similar messages [ 1684.985154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1684.995303] Lustre: Skipped 2 previous similar messages [ 1690.084968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1692.630864] Lustre: server umount lustre-MDT0000 complete [ 1695.202432] LustreError: 48116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1695.220806] LustreError: 48116:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1696.706878] LustreError: 48096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240181 with bad export cookie 4923549631510179654 [ 1696.711562] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1696.711787] LustreError: 48096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1697.319725] Lustre: server umount lustre-MDT0001 complete [ 1712.185779] Lustre: server umount lustre-OST0000 complete [ 1741.607377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1741.952433] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1742.253633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1748.458480] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1758.414060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1758.494929] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1758.559126] Lustre: 61688:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1763.854064] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1770.305308] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1773.838561] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1780.752686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1781.061534] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1781.425262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1781.437981] Lustre: Skipped 1 previous similar message [ 1787.863596] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1794.811555] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1796.745463] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1796.778151] Lustre: Skipped 23 previous similar messages [ 1796.855614] Lustre: 63916:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1808.866148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1808.868796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1808.877017] LustreError: Skipped 1 previous similar message [ 1808.882534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1808.894771] Lustre: Skipped 1 previous similar message [ 1808.912109] Lustre: Skipped 3 previous similar messages [ 1812.850430] Lustre: server umount lustre-OST0000 complete [ 1819.105104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1819.116802] Lustre: Skipped 1 previous similar message [ 1824.234053] LustreError: 62422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1824.237058] Lustre: server umount lustre-MDT0000 complete [ 1824.251036] LustreError: 62422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1828.340783] LustreError: 61679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240313 with bad export cookie 4923549631510182783 [ 1828.354746] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1828.366625] LustreError: 61679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1828.704980] Lustre: server umount lustre-MDT0001 complete [ 1839.524875] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 1841.915294] Key type lgssc unregistered [ 1842.361668] LNet: 65109:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1842.370170] LNetError: 65109:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1842.389704] LNet: Removed LNI 192.168.201.110@tcp [ 1843.159963] Key type .llcrypt unregistered [ 1843.164127] Key type ._llcrypt unregistered [ 1864.797412] Key type ._llcrypt registered [ 1864.799764] Key type .llcrypt registered [ 1864.880740] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 1866.083312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1866.103792] alg: No test for adler32 (adler32-zlib) [ 1867.117391] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 1867.362833] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 1869.008136] Key type lgssc registered [ 1869.881880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1877.114336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1877.127378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1878.549112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1882.836528] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1888.737644] LustreError: 66506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1888.758791] LustreError: 66506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1891.900870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1892.532405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1896.718382] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1902.310925] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1907.501549] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1915.060664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1915.597082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1923.168748] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1928.692827] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1937.913034] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1948.646633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1948.657733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1948.680098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1949.667732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1949.668949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1949.712479] Lustre: Skipped 1 previous similar message [ 1954.456485] Lustre: server umount lustre-MDT0000 complete [ 1954.800653] LustreError: 66501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1954.826362] LustreError: 66501:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1958.327656] LustreError: 66487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240443 with bad export cookie 2030965066318558660 [ 1958.341128] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1958.346898] LustreError: 66487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1958.653065] Lustre: server umount lustre-MDT0001 complete [ 1972.373351] Lustre: server umount lustre-OST0000 complete [ 2001.140717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2001.381602] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2001.403313] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2001.752495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2007.154587] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2015.987370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2016.145806] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2016.191497] Lustre: 71052:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2016.822387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2021.460872] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2027.373425] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2031.110472] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2036.996843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2037.150524] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2037.426545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2043.308780] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2049.522922] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2051.240083] Lustre: 73285:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2052.919219] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2058.724315] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2058.729506] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2058.739901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2058.748399] Lustre: Skipped 2 previous similar messages [ 2064.293346] Lustre: server umount lustre-OST0000 complete [ 2069.476265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2069.490175] Lustre: Skipped 1 previous similar message [ 2069.496046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2069.501397] Lustre: Skipped 3 previous similar messages [ 2074.431626] Lustre: server umount lustre-MDT0000 complete [ 2074.608857] LustreError: 71062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2074.639403] LustreError: 71062:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2078.389056] LustreError: 72692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240563 with bad export cookie 2030965066318559507 [ 2078.391303] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2078.402375] LustreError: 72692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2078.750262] Lustre: server umount lustre-MDT0001 complete [ 2089.308231] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 2091.711131] Key type lgssc unregistered [ 2092.045078] LNet: 74479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2092.049126] LNetError: 74479:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2092.069371] LNet: Removed LNI 192.168.201.110@tcp [ 2092.783419] Key type .llcrypt unregistered [ 2092.785985] Key type ._llcrypt unregistered [ 2117.457646] Key type ._llcrypt registered [ 2117.460604] Key type .llcrypt registered [ 2117.572409] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2119.275691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2119.330525] alg: No test for adler32 (adler32-zlib) [ 2120.407997] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 2120.559437] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 2122.208182] Key type lgssc registered [ 2123.780784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2132.215260] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2132.250450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2133.949206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2138.275047] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2143.205278] LustreError: 75874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2143.219775] LustreError: 75874:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2145.546750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2146.078200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2150.528929] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2156.094843] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2159.940806] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2166.170838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2166.571977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2172.540559] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2177.697310] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2183.873351] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2184.725945] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2186.778295] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2186.781563] Lustre: Skipped 1 previous similar message [ 2188.867452] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2188.872203] Lustre: Skipped 1 previous similar message [ 2193.363825] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2193.366518] Lustre: Skipped 4 previous similar messages [ 2201.795671] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2201.802871] Lustre: Skipped 10 previous similar messages [ 2218.465509] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2218.469787] Lustre: Skipped 19 previous similar messages [ 2251.181480] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2251.185165] Lustre: Skipped 37 previous similar messages [ 2275.288605] Lustre: 83628:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2277.908743] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 01:06:01 (1781240761) [ 2279.954705] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2281.792417] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 01:06:05 (1781240765) [ 2288.611769] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2288.618550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2288.653244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2291.682125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2291.684414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2291.687181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2291.692940] Lustre: Skipped 1 previous similar message [ 2293.187784] Lustre: server umount lustre-OST0000 complete [ 2299.362045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2299.376774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2299.405910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2301.922918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2301.924504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2303.248457] Lustre: server umount lustre-MDT0000 complete [ 2307.025251] LustreError: 75854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240791 with bad export cookie 16632003795409187197 [ 2307.064986] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2307.466397] Lustre: server umount lustre-MDT0001 complete [ 2319.861962] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 2322.737496] Key type lgssc unregistered [ 2323.009651] LNet: 84863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2323.023269] LNetError: 84863:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2323.053543] LNet: Removed LNI 192.168.201.110@tcp [ 2323.767552] Key type .llcrypt unregistered [ 2323.775919] Key type ._llcrypt unregistered [ 2327.800601] Key type ._llcrypt registered [ 2327.802477] Key type .llcrypt registered [ 2327.950464] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2332.647329] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules [ 2333.748924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2333.791254] alg: No test for adler32 (adler32-zlib) [ 2335.052758] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 2335.260398] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 2336.976171] Key type lgssc registered [ 2338.332542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2339.601480] LustreError: 86031:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2356.531789] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2361.705933] LustreError: 86031:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2365.470522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2365.492995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2367.133407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2371.704368] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2376.675499] LustreError: 87039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2376.687493] LustreError: 87039:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2380.291517] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 01:07:43 (1781240863) [ 2381.795888] LustreError: 87040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2384.538767] LustreError: 87071:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2384.550512] LustreError: 87071:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2384.556257] LustreError: 87071:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2384.790394] Lustre: server umount lustre-MDT0000 complete [ 2398.211558] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 2401.055644] Key type lgssc unregistered [ 2401.316488] LNet: 88530:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2401.321737] LNetError: 88530:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2401.335663] LNet: Removed LNI 192.168.201.110@tcp [ 2402.200247] Key type .llcrypt unregistered [ 2402.202542] Key type ._llcrypt unregistered [ 2421.401854] Key type ._llcrypt registered [ 2421.410728] Key type .llcrypt registered [ 2421.536819] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2422.524672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2422.573017] alg: No test for adler32 (adler32-zlib) [ 2423.822467] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 2424.030857] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 2425.712617] Key type lgssc registered [ 2426.770901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2434.789438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2434.820687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2436.546987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2441.684593] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2446.825427] LustreError: 89730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.863982] LustreError: 89730:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2451.211918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2451.906928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2456.582844] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2463.229710] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2466.835892] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2474.637679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2475.620048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2483.742962] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2491.419694] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2541.189691] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 01:10:24 (1781241024) [ 2543.181526] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2545.443267] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 01:10:28 (1781241028) [ 2553.826816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2553.835379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2553.838651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2553.848953] Lustre: Skipped 2 previous similar messages [ 2553.874382] Lustre: Skipped 2 previous similar messages [ 2556.121097] Lustre: server umount lustre-MDT0000 complete [ 2558.946563] LustreError: 89730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.955505] LustreError: 89730:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2559.577488] LustreError: 89710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241044 with bad export cookie 5685829500603046868 [ 2559.580408] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2559.587167] LustreError: 89710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2559.899460] Lustre: server umount lustre-MDT0001 complete [ 2574.429881] Lustre: server umount lustre-OST0000 complete [ 2584.505557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2584.931524] LustreError: 93668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.088374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2589.030433] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2590.182095] LustreError: 93668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2590.191358] LustreError: 93668:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2594.276375] LustreError: 93669:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2597.434858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2597.980473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2602.398655] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2608.150432] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2612.221405] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2626.923167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2627.196184] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2627.203836] LustreError: 93657:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2627.251638] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2627.261622] LustreError: 95526:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2627.271777] LustreError: 95526:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2627.280449] LustreError: 95526:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2627.351789] Lustre: server umount lustre-OST0000 complete [ 2627.357299] LustreError: 95526:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2638.609736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2638.889928] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2638.904912] LustreError: 93657:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2638.944646] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2638.965302] LustreError: 95994:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2638.980270] LustreError: 95994:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2638.986088] LustreError: 95994:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2639.111378] Lustre: server umount lustre-OST0000 complete [ 2639.127987] LustreError: 95994:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2650.961459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2651.181229] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2651.192470] Lustre: Found index 0 for lustre-OST0000, updating log [ 2651.203390] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2651.326175] Lustre: lustre-OST0000: new disk, initializing [ 2651.345405] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2651.365824] Lustre: 96528:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2651.777796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2658.327835] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2662.605386] Lustre: Failing over lustre-OST0000 [ 2662.916979] Lustre: server umount lustre-OST0000 complete [ 2668.796384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2668.850608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2668.854032] Lustre: Skipped 1 previous similar message [ 2674.148726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2674.161467] Lustre: Skipped 1 previous similar message [ 2679.268243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2679.287031] Lustre: Skipped 1 previous similar message [ 2682.338405] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2682.671111] Lustre: server umount lustre-MDT0000 complete [ 2684.385962] LustreError: 94834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2684.399361] LustreError: 94834:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2687.182879] LustreError: 93651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241172 with bad export cookie 5685829500603049612 [ 2687.182879] LustreError: 93649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241172 with bad export cookie 5685829500603049612 [ 2687.184227] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2687.197171] LustreError: 93649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2687.481317] Lustre: server umount lustre-MDT0001 complete [ 2696.733975] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 01:13:00 (1781241180) [ 2712.667890] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2722.926128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2723.292758] LustreError: 98839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2723.448650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2727.621568] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2736.613582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2741.488819] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2744.462399] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2750.882448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2751.442713] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2758.180900] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2761.702432] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2761.719951] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2761.751149] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2766.823635] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2766.842840] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2766.911551] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2767.039872] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2767.159220] Lustre: lustre-OST0001: new disk, initializing [ 2767.174807] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2767.182609] Lustre: 101105:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2767.534502] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2767.543033] Lustre: Skipped 2 previous similar messages [ 2768.571421] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:02 [ 2768.592618] Lustre: Skipped 1 previous similar message [ 2768.626644] Lustre: lustre-OST0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted. [ 2768.639703] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2768.654623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2773.866370] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2776.801890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2776.817593] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2776.975062] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2782.744848] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2789.031212] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2793.953761] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2793.971377] LustreError: Skipped 2 previous similar messages [ 2793.974289] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2793.989257] Lustre: Skipped 1 previous similar message [ 2794.014284] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2797.557581] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2797.573158] Lustre: Skipped 1 previous similar message [ 2802.624747] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.10@tcp (stopping) [ 2802.635352] Lustre: Skipped 4 previous similar messages [ 2808.289047] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2808.498634] Lustre: server umount lustre-MDT0001 complete [ 2812.859082] LustreError: 100321:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2812.885692] LustreError: 100321:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2817.299610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2817.408458] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2817.411294] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2817.457082] Lustre: 98830:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2818.997839] LustreError: 102632:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2819.003678] LustreError: 102632:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2819.009973] LustreError: 98832:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2819.111763] LustreError: 102643:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.110@tcp: cfg command failed: rc = -17 [ 2819.119315] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2819.119315] [ 2823.122305] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2823.690976] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2834.565421] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 01:15:17 (1781241317) [ 2854.181423] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2873.480197] Lustre: 104673:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2894.903956] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2898.158141] Lustre: 105807:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2987.490317] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2987.501825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2987.504480] Lustre: Skipped 2 previous similar messages [ 2987.524580] Lustre: Skipped 10 previous similar messages [ 2993.844344] Lustre: server umount lustre-MDT0000 complete [ 2997.416310] LustreError: 98820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241482 with bad export cookie 5685829500603050613 [ 2997.421093] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2997.432247] LustreError: 98820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2997.668200] Lustre: server umount lustre-MDT0001 complete [ 3011.134919] Lustre: server umount lustre-OST0000 complete [ 3013.088616] Lustre: 89041:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781241482/real 1781241482] req@ffff9f43d3be8f00 x1867766466707328/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781241498 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3013.130585] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3013.151897] Lustre: Skipped 1 previous similar message [ 3015.588973] Lustre: server umount lustre-OST0001 complete [ 3021.720641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3033.068821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3041.717398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3052.689212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3071.895716] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3082.441597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3082.459899] Lustre: lustre-MDT0000: reset Object Index mappings [ 3082.894365] LustreError: 109744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3082.908663] LustreError: 109744:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3082.993281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3083.000798] Lustre: Skipped 1 previous similar message [ 3087.528311] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3095.803983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3095.826996] Lustre: lustre-MDT0001: reset Object Index mappings [ 3101.010290] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3104.053482] Lustre: 110857:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3111.075570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3117.302598] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3121.984829] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3125.435677] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3128.027585] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3128.029301] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3128.244486] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3132.903824] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3142.021571] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3145.886438] Lustre: 112702:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3201.467391] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 01:21:24 (1781241684) [ 3229.665894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3229.680887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3229.703097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3229.707255] Lustre: Skipped 4 previous similar messages [ 3233.817974] Lustre: server umount lustre-MDT0000 complete [ 3235.814707] LustreError: 109740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3235.866696] LustreError: 109740:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3237.879274] LustreError: 109725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241722 with bad export cookie 5685829500603097751 [ 3237.886087] LustreError: 109725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3237.892135] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3252.192531] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3252.522773] Lustre: server umount lustre-MDT0001 complete [ 3262.653578] Lustre: server umount lustre-OST0000 complete [ 3272.537722] Lustre: server umount lustre-OST0001 complete [ 3279.903956] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 3292.182725] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3331.741726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3332.024208] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3332.058491] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3332.118375] Lustre: lustre-MDT0000: new disk, initializing [ 3332.179080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3332.185221] Lustre: Skipped 3 previous similar messages [ 3332.203848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3337.198784] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3344.263527] Lustre: server umount lustre-MDT0000 complete [ 3353.230692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3353.424725] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3353.431567] Lustre: 116923:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3353.438624] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3357.455169] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3362.976370] Lustre: server umount lustre-MDT0000 complete [ 3372.804550] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 01:24:16 (1781241856) [ 3374.592054] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3376.834495] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 01:24:20 (1781241860) [ 3396.278267] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3407.081864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3412.591461] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3421.356127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3421.442249] Lustre: 118941:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3421.493401] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3421.496952] Lustre: Skipped 1 previous similar message [ 3421.552271] Lustre: lustre-MDT0001: new disk, initializing [ 3421.638385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3421.662627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3425.938563] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3431.323455] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3439.595627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3439.885426] Lustre: lustre-OST0000: new disk, initializing [ 3439.890899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3439.904139] Lustre: 120483:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3441.074105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3441.094551] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3441.303867] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3448.561862] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3460.389341] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3460.560720] Lustre: lustre-OST0001: new disk, initializing [ 3460.564528] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3460.569771] Lustre: 121338:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3462.567067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3462.583060] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3462.697539] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3467.851736] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3479.200098] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3483.494826] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3866.751254] Lustre: 127742:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3866.760591] Lustre: 127742:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3866.780914] Lustre: 127742:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3866.794223] Lustre: 127742:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3866.799678] Lustre: 127742:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3866.806928] Lustre: 127742:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4168.803504] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:37:30 (1781242650) [ 4184.550820] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4184.560617] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4184.569279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4184.569287] Lustre: Skipped 12 previous similar messages [ 4184.589937] Lustre: Skipped 8 previous similar messages [ 4186.345114] Lustre: server umount lustre-MDT0000 complete [ 4189.667766] LustreError: 120476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4189.694117] LustreError: 120476:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 4190.875454] LustreError: 120482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242675 with bad export cookie 5685829500603166183 [ 4190.892400] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4190.900518] LustreError: 120482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4194.786446] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4194.789119] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4194.815408] Lustre: Skipped 1 previous similar message [ 4194.832033] Lustre: Skipped 4 previous similar messages [ 4197.427194] Lustre: server umount lustre-MDT0001 complete [ 4207.716521] Lustre: server umount lustre-OST0000 complete [ 4218.149660] Lustre: server umount lustre-OST0001 complete [ 4225.782993] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 4237.753248] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 4274.441711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4274.735357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4274.778857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4274.844165] Lustre: lustre-MDT0000: new disk, initializing [ 4274.934465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4274.946517] Lustre: Skipped 5 previous similar messages [ 4274.973043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4279.263458] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4290.031712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4290.159907] Lustre: 130769:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4290.206364] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4290.210349] Lustre: Skipped 1 previous similar message [ 4290.308225] Lustre: lustre-MDT0001: new disk, initializing [ 4290.459345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4290.471233] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4295.679458] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4303.913957] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4308.224562] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4315.650523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4315.998266] Lustre: lustre-OST0000: new disk, initializing [ 4316.001312] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4316.008687] Lustre: 132513:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4316.099719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4316.110827] Lustre: Skipped 1 previous similar message [ 4318.019224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4318.029645] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4318.180315] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4322.814745] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4330.413727] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4339.824340] Lustre: lustre-MDD0000: changelog on [ 4342.247544] Lustre: lustre-MDD0001: changelog on [ 6291.129284] Lustre: lustre-MDD0001: changelog off [ 6293.011934] Lustre: 130778:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6294.115627] Lustre: lustre-MDD0000: changelog off [ 6299.620316] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6299.623399] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6299.627706] LustreError: Skipped 1 previous similar message [ 6299.629577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6299.629583] Lustre: Skipped 2 previous similar messages [ 6299.650272] Lustre: Skipped 1 previous similar message [ 6304.746785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6304.754516] Lustre: Skipped 2 previous similar messages [ 6305.522055] Lustre: server umount lustre-OST0000 complete [ 6309.857676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6309.868380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6309.871452] Lustre: Skipped 1 previous similar message [ 6315.583305] Lustre: server umount lustre-MDT0000 complete [ 6319.551026] LustreError: 130763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244804 with bad export cookie 5685829500603986702 [ 6319.556356] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6319.559395] LustreError: 130763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6320.010498] Lustre: server umount lustre-MDT0001 complete [ 6330.964774] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 6335.010211] Key type lgssc unregistered [ 6335.380568] LNet: 141530:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6335.390425] LNetError: 141530:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6336.427651] LNet: Removed LNI 192.168.201.110@tcp [ 6337.611186] Key type .llcrypt unregistered [ 6337.615460] Key type ._llcrypt unregistered [ 6364.200295] Key type ._llcrypt registered [ 6364.203919] Key type .llcrypt registered [ 6364.376160] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 6385.391418] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 6387.163520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6387.406402] alg: No test for adler32 (adler32-zlib) [ 6388.553453] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 6388.773185] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 6390.432219] Key type lgssc registered [ 6391.778211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6430.664208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6430.698390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6431.971328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6431.994790] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6432.059904] Lustre: lustre-MDT0000: new disk, initializing [ 6432.154652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6432.167591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6437.127706] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6448.866281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6449.023392] Lustre: 144701:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6449.087887] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6449.092914] Lustre: Skipped 1 previous similar message [ 6449.145203] Lustre: lustre-MDT0001: new disk, initializing [ 6449.255647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6449.305958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6449.316993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6453.519222] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6462.136177] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6466.736179] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6475.093938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6475.570101] Lustre: lustre-OST0000: new disk, initializing [ 6475.579938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6475.593749] Lustre: 146444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6475.759589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6482.019307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6482.038873] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6482.248729] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6483.500644] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6491.406988] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6499.130799] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_module obdecho/obdecho [ 6501.676748] LustreError: 147317:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6501.688728] LustreError: 147317:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6504.213955] LustreError: 147413:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 6518.256845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6518.261741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6518.281536] Lustre: Skipped 2 previous similar messages [ 6518.285160] Lustre: Skipped 1 previous similar message [ 6523.490826] Lustre: server umount lustre-MDT0000 complete [ 6527.562263] LustreError: 144693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245012 with bad export cookie 4244573807961963929 [ 6527.567840] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6527.572839] LustreError: 144693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6527.941980] Lustre: server umount lustre-MDT0001 complete [ 6542.867929] Lustre: server umount lustre-OST0000 complete [ 6566.878141] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 6580.003909] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 6623.559751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6623.935981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6623.961706] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6624.044747] Lustre: lustre-MDT0000: new disk, initializing [ 6624.183988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6624.221917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6630.081018] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6645.728623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6646.049644] Lustre: 151100:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6646.169479] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6646.173886] Lustre: Skipped 1 previous similar message [ 6646.322559] Lustre: lustre-MDT0001: new disk, initializing [ 6646.578930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6646.642728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6646.664279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6653.427273] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6663.220213] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6666.846302] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6675.663682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6676.041576] Lustre: lustre-OST0000: new disk, initializing [ 6676.048763] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6676.061842] Lustre: 152843:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6676.185877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6682.736991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6682.765226] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6682.962976] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6683.647349] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6692.625889] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6698.051203] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6698.371375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6702.974823] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6703.147119] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6708.717805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6708.718292] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6708.732814] LustreError: Skipped 1 previous similar message [ 6708.782972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6708.807940] Lustre: Skipped 2 previous similar messages [ 6711.609389] Lustre: server umount lustre-OST0000 complete [ 6718.433142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6718.436237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6718.458255] Lustre: Skipped 1 previous similar message [ 6718.462779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6722.335504] Lustre: server umount lustre-MDT0000 complete [ 6724.069352] LustreError: 151111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6724.108945] LustreError: 151111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6726.487276] LustreError: 151090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245211 with bad export cookie 4244573807961964888 [ 6726.495864] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6726.497833] LustreError: 151090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6726.829324] Lustre: server umount lustre-MDT0001 complete [ 6731.526334] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 02:20:14 (1781245214) [ 6734.082393] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6736.397208] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 02:20:19 (1781245219) [ 6745.966101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6746.413737] LustreError: 155108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6746.476400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6750.644629] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6751.716225] LustreError: 155109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6756.835647] LustreError: 155108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6759.104553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6763.620829] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6769.040772] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6774.181649] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6784.185478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6784.700748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6784.714489] Lustre: Skipped 1 previous similar message [ 6791.831524] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6799.822993] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6810.582870] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.10@tcp (stopping) [ 6810.595296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6810.597360] Lustre: Skipped 1 previous similar message [ 6810.615840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6810.644931] Lustre: Skipped 1 previous similar message [ 6811.456062] Lustre: server umount lustre-MDT0000 complete [ 6812.647741] LustreError: 155819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6812.657874] LustreError: 155819:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6815.654144] LustreError: 155089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245300 with bad export cookie 4244573807961965770 [ 6815.662128] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6815.673533] LustreError: 155089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6815.735634] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.10@tcp (stopping) [ 6815.747788] Lustre: Skipped 1 previous similar message [ 6817.765442] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6817.788474] Lustre: Skipped 2 previous similar messages [ 6820.785856] LustreError: 155103:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6820.824180] LustreError: 155103:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6822.227242] Lustre: server umount lustre-MDT0001 complete [ 6832.101255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6848.232163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6848.474948] LustreError: 158615:0:(llog.c:1646:llog_backup()) MGC192.168.201.110@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6848.485404] Lustre: 158615:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.110@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6850.530210] LustreError: 158620:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6885.348838] LustreError: 158622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6885.381992] LustreError: 158622:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 20 previous similar messages [ 6909.408339] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6909.414522] Lustre: 158615:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.201.110@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6950.417995] LustreError: 158621:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6950.420248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6950.434589] LustreError: 158621:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 42 previous similar messages [ 6955.502731] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6955.794166] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6965.143352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6981.092496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6985.198068] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3ae7c163be801e07 [ 6985.220657] Lustre: MGC192.168.201.110@tcp: Connection restored to 0@lo (at 0@lo) [ 6989.932662] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6990.831619] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6995.910362] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6997.520704] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7001.397755] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7003.293049] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7031.130399] Lustre: server umount lustre-MDT0000 complete [ 7031.778300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7031.783699] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7031.802706] Lustre: Skipped 2 previous similar messages [ 7035.136750] LustreError: 155088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245520 with bad export cookie 4244573807961972231 [ 7035.139476] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7035.156279] LustreError: 155088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7035.592400] Lustre: server umount lustre-MDT0001 complete [ 7050.255259] Lustre: server umount lustre-OST0000 complete [ 7074.482309] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 7086.510991] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 7123.496342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7123.843817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7123.873579] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7123.952890] Lustre: lustre-MDT0000: new disk, initializing [ 7124.048407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7124.056894] Lustre: Skipped 1 previous similar message [ 7124.076215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7128.512823] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7139.836414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7139.967723] Lustre: 163467:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7140.298477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7144.547795] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7152.463784] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7156.057089] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7163.084744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7163.407607] Lustre: lustre-OST0000: new disk, initializing [ 7163.410832] Lustre: Skipped 1 previous similar message [ 7163.415311] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7163.420857] Lustre: Skipped 2 previous similar messages [ 7163.431368] Lustre: 165206:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7169.353701] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7172.694105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7172.705473] Lustre: Skipped 1 previous similar message [ 7172.708890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7172.835490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7176.946103] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7181.501531] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7181.677515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7185.545638] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7185.715962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7188.453071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7188.457939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7188.477398] Lustre: Skipped 1 previous similar message [ 7188.500619] Lustre: Skipped 4 previous similar messages [ 7193.533212] Lustre: server umount lustre-OST0000 complete [ 7203.812617] LustreError: 163479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7203.826192] LustreError: 163479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 33 previous similar messages [ 7203.936050] Lustre: server umount lustre-MDT0000 complete [ 7207.456466] LustreError: 163459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245692 with bad export cookie 4244573807961989066 [ 7207.476298] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7207.822355] Lustre: server umount lustre-MDT0001 complete [ 7216.293896] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 02:28:19 (1781245699) [ 7222.841950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7227.760928] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7236.356696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7242.924572] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7253.924302] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7260.261172] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7270.527698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7277.651702] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7284.303262] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7307.196252] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 02:29:50 (1781245790) [ 7312.361732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7312.373185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7312.384094] Lustre: Skipped 2 previous similar messages [ 7312.400217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7312.405835] Lustre: Skipped 2 previous similar messages [ 7318.015600] Lustre: server umount lustre-OST0000 complete [ 7328.903788] Lustre: server umount lustre-MDT0000 complete [ 7333.505045] LustreError: 167450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245818 with bad export cookie 4244573807961989948 [ 7333.518884] LustreError: 167450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7333.525260] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7333.795746] Lustre: server umount lustre-MDT0001 complete [ 7346.868143] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 7350.548753] Key type lgssc unregistered [ 7350.926675] LNet: 170869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7350.930536] LNetError: 170869:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7350.941523] LNet: Removed LNI 192.168.201.110@tcp [ 7352.007136] Key type .llcrypt unregistered [ 7352.011449] Key type ._llcrypt unregistered [ 7372.544288] Key type ._llcrypt registered [ 7372.547780] Key type .llcrypt registered [ 7372.669151] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 7374.092975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7374.130915] alg: No test for adler32 (adler32-zlib) [ 7375.135423] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 7375.338903] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 7376.979187] Key type lgssc registered [ 7378.077265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7386.482343] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7386.507933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7402.081952] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7407.264392] LustreError: 172097:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.110@tcp: failed processing log, type 4: rc = -110 [ 7412.390664] LustreError: 172097:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7412.414638] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7412.421696] LustreError: MGC192.168.201.110@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7412.430344] LustreError: 172097:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7412.437724] LustreError: 172097:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 7412.444050] LustreError: 172097:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 7412.533091] Lustre: server umount lustre-OST0000 complete [ 7412.535809] LustreError: 172097:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7421.744459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7423.150097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7427.699134] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7433.711701] LustreError: 172595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7433.748602] LustreError: 172595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7436.598456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7436.941946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7442.356657] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7448.767128] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7453.130290] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7460.187211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7460.400903] LustreError: 174185:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7460.411819] Lustre: 174185:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.110@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7460.577092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7466.757739] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7473.945158] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7477.729919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7477.747211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7477.770960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7479.266131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7479.268495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7479.289531] Lustre: Skipped 1 previous similar message [ 7479.328358] Lustre: Skipped 1 previous similar message [ 7482.952675] Lustre: server umount lustre-MDT0000 complete [ 7484.410949] LustreError: 172596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7487.489285] LustreError: 172576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245972 with bad export cookie 10728610448960268394 [ 7487.494417] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7487.501330] LustreError: 172576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7487.922645] Lustre: server umount lustre-MDT0001 complete [ 7502.854058] Lustre: server umount lustre-OST0000 complete [ 7512.308703] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 7522.734846] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 7567.872591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7568.564849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7568.654762] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7568.876297] Lustre: lustre-MDT0000: new disk, initializing [ 7569.203604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7569.244312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7574.654828] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7586.247634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7586.353394] Lustre: 177565:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7586.406573] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7586.410494] Lustre: Skipped 1 previous similar message [ 7586.487691] Lustre: lustre-MDT0001: new disk, initializing [ 7586.606763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7586.650531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7586.662798] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7591.898717] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7600.443077] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7605.292407] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7613.571042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7613.810743] Lustre: lustre-OST0000: new disk, initializing [ 7613.814131] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7613.819458] Lustre: 179306:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7613.917433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7619.788786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7619.816414] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7620.014272] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7623.066957] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7632.350755] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7637.955237] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7638.150299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7642.981133] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7643.151092] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7645.674468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7645.677832] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7645.695367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7645.695384] Lustre: Skipped 1 previous similar message [ 7645.743657] LustreError: Skipped 1 previous similar message [ 7650.798105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7650.800947] Lustre: Skipped 2 previous similar messages [ 7651.649096] Lustre: server umount lustre-OST0000 complete [ 7655.907970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7655.912840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7655.924723] Lustre: Skipped 2 previous similar messages [ 7655.935280] Lustre: Skipped 1 previous similar message [ 7666.146935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7666.152293] Lustre: Skipped 3 previous similar messages [ 7670.240429] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7670.585841] Lustre: server umount lustre-MDT0000 complete [ 7671.271455] LustreError: 177576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7671.287048] LustreError: 177576:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7675.361949] LustreError: 177575:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7675.381273] LustreError: 177575:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7676.191124] LustreError: 179305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781246161 with bad export cookie 10728610448960269122 [ 7676.193507] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7676.207046] LustreError: 179305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7676.735800] Lustre: server umount lustre-MDT0001 complete [ 7688.649359] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 02:36:11 (1781246171) [ 7708.000367] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 7710.591900] Key type lgssc unregistered [ 7710.837722] LNet: 182092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7710.849088] LNetError: 182092:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7710.877310] LNet: Removed LNI 192.168.201.110@tcp [ 7711.826626] Key type .llcrypt unregistered [ 7711.834206] Key type ._llcrypt unregistered [ 7733.659764] Key type ._llcrypt registered [ 7733.662788] Key type .llcrypt registered [ 7733.839872] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 7735.431296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7735.516082] alg: No test for adler32 (adler32-zlib) [ 7736.762811] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 7737.010819] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 7738.648466] Key type lgssc registered [ 7740.108756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7747.847000] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7747.875164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7749.806790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7754.755565] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7759.330301] LustreError: 183294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7759.354035] LustreError: 183294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7763.496188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7763.947654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7768.987786] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7774.848549] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7778.904464] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7787.137141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7787.433626] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7787.435937] LustreError: 184931:0:(llog.c:1687:llog_backup()) MGC192.168.201.110@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7787.444682] Lustre: 184931:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.110@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7787.610801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7793.244655] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7799.759404] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7804.900814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7804.904201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7804.931609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7805.413149] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7805.422287] Lustre: Skipped 1 previous similar message [ 7805.435337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7805.437382] Lustre: Skipped 2 previous similar messages [ 7810.531871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7810.538100] Lustre: Skipped 2 previous similar messages [ 7815.654294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7815.660058] Lustre: Skipped 2 previous similar messages [ 7816.160184] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7816.483817] Lustre: server umount lustre-MDT0000 complete [ 7820.691306] LustreError: 183274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781246305 with bad export cookie 15465721601755923954 [ 7820.739591] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7820.769217] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7820.772502] LustreError: 183291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7820.826791] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7821.205603] Lustre: server umount lustre-MDT0001 complete [ 7828.115619] Lustre: server umount lustre-OST0000 complete [ 7839.156641] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 7849.178268] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 7891.960406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7892.145365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7892.171498] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7892.236642] Lustre: lustre-MDT0000: new disk, initializing [ 7892.387371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7892.408498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7897.758073] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7910.908807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7911.127378] Lustre: 188310:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7911.218321] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7911.222812] Lustre: Skipped 1 previous similar message [ 7911.233433] LustreError: 189074:0:(llog.c:1687:llog_backup()) MGC192.168.201.110@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7911.245674] LustreError: 189074:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7911.250542] Lustre: 189074:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.110@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7911.293152] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7911.301426] Lustre: Skipped 1 previous similar message [ 7911.430920] Lustre: lustre-MDT0001: new disk, initializing [ 7911.539979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7911.593119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7911.617202] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7916.586625] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7924.811487] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7928.553586] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7935.555182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7935.844707] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7935.847745] Lustre: Skipped 1 previous similar message [ 7935.852446] LustreError: 190027:0:(llog.c:1687:llog_backup()) MGC192.168.201.110@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7935.867852] LustreError: 190027:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7935.878503] Lustre: 190027:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.110@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7935.885856] Lustre: 190027:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7935.926700] Lustre: lustre-OST0000: new disk, initializing [ 7935.933042] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7935.941049] Lustre: 190051:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7936.074928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7941.604262] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7946.311183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7946.322177] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7946.444221] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7949.380968] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7954.250752] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7954.440158] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7959.377475] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7959.681397] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7962.083264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7962.101627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7962.122122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7962.132114] Lustre: Skipped 1 previous similar message [ 7968.276137] Lustre: server umount lustre-OST0000 complete [ 7972.837300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7972.853232] LustreError: Skipped 1 previous similar message [ 7972.857357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7972.893052] Lustre: Skipped 1 previous similar message [ 7979.272822] Lustre: server umount lustre-MDT0000 complete [ 7982.566326] LustreError: 188316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7982.574942] LustreError: 188316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7985.615293] LustreError: 188300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781246470 with bad export cookie 15465721601755924675 [ 7985.621707] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7985.634757] LustreError: 188300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7985.994037] Lustre: server umount lustre-MDT0001 complete [ 7998.440585] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 02:41:21 (1781246481) [ 8018.813556] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 8037.290823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8037.803530] LustreError: 193017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8037.879722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8042.979368] LustreError: 193018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8043.880957] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8052.193352] LustreError: 193017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8052.209317] LustreError: 193017:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8055.862530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8061.056169] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8064.452729] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8073.111386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8073.617341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8073.625712] Lustre: Skipped 1 previous similar message [ 8080.980483] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8094.080530] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8094.225702] Lustre: lustre-OST0001: new disk, initializing [ 8094.227694] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8094.232155] Lustre: 195277:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8102.718528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8102.745633] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8103.057620] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8106.386312] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8120.666901] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8129.020925] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8131.841252] Lustre: DEBUG MARKER: ADD OST3 [ 8138.472629] loop: module loaded [ 8138.636540] loop0: detected capacity change from 0 to 204800000 [ 8138.947083] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8138.964289] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8139.000900] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8146.995055] loop0: detected capacity change from 0 to 204800000 [ 8147.055851] Lustre: DEBUG MARKER: STOP OST3 [ 8153.964127] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8154.223684] Lustre: lustre-OST0003: new disk, initializing [ 8154.231292] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8154.244601] Lustre: 197334:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8154.371280] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8154.378927] Lustre: Skipped 1 previous similar message [ 8155.365731] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8155.518821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8155.525798] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8155.581905] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8156.826659] Lustre: Failing over lustre-OST0003 [ 8156.993693] Lustre: server umount lustre-OST0003 complete [ 8157.413510] LustreError: 197415:0:(osp_precreate.c:1281:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8160.737076] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8160.753643] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8160.770433] LustreError: 194471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8160.773234] Lustre: Skipped 2 previous similar messages [ 8160.817664] LustreError: 194471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8161.690348] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8171.716505] Lustre: DEBUG MARKER: START OST3 again [ 8180.025708] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8180.382707] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8181.425244] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8181.510609] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8181.514261] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8189.051567] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8214.348869] Lustre: Failing over lustre-OST0003 [ 8214.554314] Lustre: server umount lustre-OST0003 complete [ 8216.570814] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8216.594546] LustreError: 194468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8216.604411] Lustre: Skipped 1 previous similar message [ 8216.620202] LustreError: 194468:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 8235.493178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8235.504231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8235.513858] Lustre: Skipped 5 previous similar messages [ 8249.312434] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8249.698480] Lustre: server umount lustre-MDT0000 complete [ 8253.858055] LustreError: 192998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781246738 with bad export cookie 15465721601755925557 [ 8253.872969] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8253.883093] LustreError: 192998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8257.506729] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8257.525988] Lustre: Skipped 4 previous similar messages [ 8268.256168] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8268.694879] Lustre: server umount lustre-MDT0001 complete [ 8273.502242] Lustre: server umount lustre-OST0000 complete [ 8277.458586] Lustre: server umount lustre-OST0001 complete [ 8284.720402] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 8294.920608] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 8339.568623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8339.851316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8339.892439] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8340.015645] Lustre: lustre-MDT0000: new disk, initializing [ 8340.141893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8340.150725] Lustre: Skipped 1 previous similar message [ 8340.177847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8345.499730] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8358.855421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8359.027381] Lustre: 202657:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8359.125136] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8359.128950] Lustre: Skipped 1 previous similar message [ 8359.181580] Lustre: lustre-MDT0001: new disk, initializing [ 8359.326885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8359.336867] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8364.673891] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8373.497754] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8377.546063] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8385.039930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8385.396621] Lustre: 204400:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8386.717178] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8386.908853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8396.100253] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8408.565485] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8416.011509] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8416.335218] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8422.226873] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8422.523564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8428.014804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8428.027926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8428.030424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8428.040107] Lustre: Skipped 2 previous similar messages [ 8428.055457] Lustre: Skipped 19 previous similar messages [ 8430.886619] Lustre: server umount lustre-OST0000 complete [ 8442.562500] Lustre: server umount lustre-MDT0000 complete [ 8443.364614] LustreError: 203508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8443.402822] LustreError: 203508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 22 previous similar messages [ 8447.133696] LustreError: 202650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781246932 with bad export cookie 15465721601755927853 [ 8447.147544] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8447.589644] Lustre: server umount lustre-MDT0001 complete [ 8451.283670] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 02:48:55 (1781246935) [ 8471.758548] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 8484.063919] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 8530.029981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8530.323407] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8530.373039] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8530.377968] Lustre: Skipped 1 previous similar message [ 8530.493950] Lustre: lustre-MDT0000: new disk, initializing [ 8530.499043] Lustre: Skipped 1 previous similar message [ 8530.647823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8530.660452] Lustre: Skipped 1 previous similar message [ 8535.713268] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8546.904883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8547.042826] Lustre: 208835:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8547.445290] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8552.187927] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8561.750307] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8566.074841] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8573.661341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8573.961687] Lustre: 210576:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8575.732417] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8582.617686] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8593.159121] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8598.927550] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8599.161879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8603.991375] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8604.184202] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8606.690036] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8606.703174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8606.709079] Lustre: Skipped 3 previous similar messages [ 8606.709707] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8606.709711] LustreError: Skipped 2 previous similar messages [ 8606.738727] Lustre: Skipped 4 previous similar messages [ 8612.962641] Lustre: server umount lustre-OST0000 complete [ 8622.892892] Lustre: server umount lustre-MDT0000 complete [ 8627.175614] LustreError: 208845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8627.193318] LustreError: 208845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8627.216852] LustreError: 208827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247112 with bad export cookie 15465721601755928735 [ 8627.229315] LustreError: 208827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8627.236196] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8628.025393] Lustre: server umount lustre-MDT0001 complete [ 8641.265754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8642.258945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8642.273842] Lustre: Skipped 5 previous similar messages [ 8649.193801] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8659.831428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8665.139748] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8671.493237] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8675.522024] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8685.294772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8693.754872] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8701.638804] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8713.699506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8718.510385] Lustre: server umount lustre-OST0000 complete [ 8737.248129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8737.566491] Lustre: server umount lustre-MDT0000 complete [ 8741.799835] LustreError: 212628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247226 with bad export cookie 15465721601755929617 [ 8741.820733] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8741.824729] LustreError: 212628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8742.091938] Lustre: server umount lustre-MDT0001 complete [ 8844.779259] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 8848.000543] Key type lgssc unregistered [ 8848.287562] LNet: 215958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8848.292895] LNetError: 215958:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8848.306694] LNet: Removed LNI 192.168.201.110@tcp [ 8849.293289] Key type .llcrypt unregistered [ 8849.295674] Key type ._llcrypt unregistered [ 8873.850712] Key type ._llcrypt registered [ 8873.852391] Key type .llcrypt registered [ 8873.961876] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 8888.383504] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 8889.322325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8889.385397] alg: No test for adler32 (adler32-zlib) [ 8890.441850] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 8890.682062] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 8892.321056] Key type lgssc registered [ 8893.425494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8925.924787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8925.948639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8927.155615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8927.177494] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8927.253637] Lustre: lustre-MDT0000: new disk, initializing [ 8927.391844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8927.416205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8930.889741] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8939.884846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8939.970710] Lustre: 219276:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8940.032683] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8940.037925] Lustre: Skipped 1 previous similar message [ 8940.108779] Lustre: lustre-MDT0001: new disk, initializing [ 8940.194463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8940.239868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8940.253852] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8944.443949] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8950.585511] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8953.987969] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8959.529546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8959.733711] Lustre: lustre-OST0000: new disk, initializing [ 8959.736668] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8959.739766] Lustre: 221017:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8959.802661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8964.434929] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8968.239649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8968.248483] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8968.368322] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8970.559305] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8973.937909] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8974.073409] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8977.344894] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8977.491772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8978.912858] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8978.917165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8978.925220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8983.521488] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8983.524710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8983.547452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8989.156393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8989.161636] Lustre: Skipped 3 previous similar messages [ 8993.250195] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8993.440614] Lustre: server umount lustre-OST0000 complete [ 8996.836336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8996.843911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8996.864076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8999.395150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9002.552767] Lustre: server umount lustre-MDT0000 complete [ 9005.600750] LustreError: 220052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247490 with bad export cookie 6711881919081079148 [ 9005.606113] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9005.607327] LustreError: 220052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9005.806209] Lustre: server umount lustre-MDT0001 complete [ 9020.038853] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 9027.900291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9028.117962] LustreError: 223785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9028.130228] LustreError: 223785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9028.171331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9031.174841] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9033.187393] LustreError: 223786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9037.721280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9037.929608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9040.842656] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9042.880094] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9047.459353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9047.661412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9051.917188] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9057.830560] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9057.971384] Lustre: lustre-OST0001: new disk, initializing [ 9057.973811] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9057.977482] Lustre: 226040:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9062.060629] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9066.016528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9066.023658] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9066.084446] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9071.432588] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9082.107358] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9090.033970] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 02:59:34 (1781247574) [ 9091.434768] Lustre: server umount lustre-MDT0000 complete [ 9091.560388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9091.571938] LustreError: 226608:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9091.581434] Lustre: Skipped 2 previous similar messages [ 9091.601352] LustreError: 226608:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9093.838167] LustreError: 223765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247578 with bad export cookie 6711881919081080030 [ 9093.842776] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9093.845811] LustreError: 223765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9094.081409] Lustre: server umount lustre-MDT0001 complete [ 9106.996241] Lustre: server umount lustre-OST0000 complete [ 9119.740684] Lustre: server umount lustre-OST0001 complete [ 9124.608668] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 9131.454336] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 9154.201526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9154.333834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9154.348523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9154.390068] Lustre: lustre-MDT0000: new disk, initializing [ 9154.453454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9154.455321] Lustre: Skipped 1 previous similar message [ 9154.462169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9156.601780] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9163.228667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9163.303797] Lustre: 229932:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9163.342550] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9163.347368] Lustre: Skipped 1 previous similar message [ 9163.412687] Lustre: lustre-MDT0001: new disk, initializing [ 9163.485233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9163.491508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9165.579421] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9170.683720] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9173.288991] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9177.782348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9178.038645] Lustre: 231684:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9179.873251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9179.878536] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9179.946679] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9182.310137] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9187.037212] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9189.289976] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9189.433763] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9191.925233] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9192.059651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9195.492222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9195.493253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9195.503937] Lustre: Skipped 1 previous similar message [ 9195.506085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9195.506093] Lustre: Skipped 2 previous similar messages [ 9199.304271] Lustre: server umount lustre-OST0000 complete [ 9204.705280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9204.711190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9204.714592] Lustre: Skipped 1 previous similar message [ 9207.746795] Lustre: server umount lustre-MDT0000 complete [ 9209.740602] LustreError: 229925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247694 with bad export cookie 6711881919081082158 [ 9209.741651] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9209.745692] LustreError: 229925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9209.923500] Lustre: server umount lustre-MDT0001 complete [ 9214.690949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9214.990218] LustreError: 233750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9215.040589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9215.044016] Lustre: Skipped 2 previous similar messages [ 9216.884781] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9220.066433] LustreError: 233750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9220.072057] LustreError: 233750:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9221.344845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9223.909810] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9227.264518] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9229.439176] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9233.260638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9236.976215] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9239.816280] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9388.003676] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9388.008949] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9388.018018] Lustre: Skipped 3 previous similar messages [ 9388.023193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9388.027242] Lustre: Skipped 2 previous similar messages [ 9391.594362] Lustre: server umount lustre-OST0000 complete [ 9399.720666] Lustre: server umount lustre-MDT0000 complete [ 9400.290497] LustreError: 233751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9400.298533] LustreError: 233751:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9401.540449] LustreError: 233731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247886 with bad export cookie 6711881919081083040 [ 9401.543490] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9401.545462] LustreError: 233731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9401.696055] Lustre: server umount lustre-MDT0001 complete [ 9507.531530] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 9510.288495] Key type lgssc unregistered [ 9510.528980] LNet: 237004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9510.538109] LNetError: 237004:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9510.561579] LNet: Removed LNI 192.168.201.110@tcp [ 9511.212315] Key type .llcrypt unregistered [ 9511.214081] Key type ._llcrypt unregistered [ 9535.452847] Key type ._llcrypt registered [ 9535.455729] Key type .llcrypt registered [ 9535.554879] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 9549.379766] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 9550.269108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9550.392079] alg: No test for adler32 (adler32-zlib) [ 9551.382595] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 9551.587390] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 9553.232186] Key type lgssc registered [ 9554.126231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9599.414185] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 9609.426661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9609.443289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9610.723222] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9610.760351] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9610.834301] Lustre: lustre-MDT0000: new disk, initializing [ 9610.898162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9610.912206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9614.553747] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9624.012910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9624.119910] Lustre: 241022:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9624.159553] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9624.163607] Lustre: Skipped 1 previous similar message [ 9624.216913] Lustre: lustre-MDT0001: new disk, initializing [ 9624.295777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9624.334074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9624.338826] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9628.309855] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9632.495469] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9638.738182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9639.034391] Lustre: lustre-OST0000: new disk, initializing [ 9639.037754] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9639.042817] Lustre: 242620:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9639.122562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9640.664075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9640.675204] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9640.870220] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9644.259773] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9653.961213] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9654.141684] Lustre: lustre-OST0001: new disk, initializing [ 9654.147223] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9654.156407] Lustre: 243473:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9654.296450] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9660.664262] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9662.080474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9662.096262] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9662.218677] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9672.325477] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9684.656424] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9696.228812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9696.235699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9696.257087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9698.274203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9698.278360] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9698.283594] Lustre: Skipped 2 previous similar messages [ 9702.167260] Lustre: server umount lustre-MDT0000 complete [ 9705.553169] LustreError: 241014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248190 with bad export cookie 6029349204830753404 [ 9705.564192] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9705.569891] LustreError: 241014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9705.924673] Lustre: server umount lustre-MDT0001 complete [ 9719.637346] Lustre: server umount lustre-OST0000 complete [ 9732.469540] Lustre: server umount lustre-OST0001 complete [ 9747.078045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9756.639771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9765.447360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9776.424583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9812.467836] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 9828.131828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9828.156222] Lustre: lustre-MDT0000: reset Object Index mappings [ 9828.825311] LustreError: 248473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9828.846709] LustreError: 248473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9828.965907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9833.959225] LustreError: 248474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9836.186472] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9839.078420] LustreError: 248473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9844.204370] LustreError: 248474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9845.816704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9845.848902] Lustre: lustre-MDT0001: reset Object Index mappings [ 9846.207801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9851.411881] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9853.897963] Lustre: 249581:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9861.773138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9862.265576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9868.167252] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9871.483041] LustreError: 249934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9874.612490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9877.034696] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9882.631567] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9883.524926] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9892.483637] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9896.784684] Lustre: 251418:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9902.570729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9902.574283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9902.598293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9902.607148] Lustre: Skipped 3 previous similar messages [ 9904.045051] Lustre: server umount lustre-MDT0000 complete [ 9907.526659] LustreError: 249958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248392 with bad export cookie 6029349204830760201 [ 9907.537360] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9907.964806] Lustre: server umount lustre-MDT0001 complete [ 9922.260787] Lustre: server umount lustre-OST0000 complete [ 9924.394752] Lustre: 238544:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781248393/real 1781248393] req@ffff9f44e19583c0 x1867773940723456/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781248409 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9924.457514] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9924.471773] Lustre: Skipped 3 previous similar messages [ 9926.959334] Lustre: server umount lustre-OST0001 complete [ 9946.959280] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:13:50 (1781248430) [ 9964.252869] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 9975.773706] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [10016.090305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10016.302963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10016.327706] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10016.398156] Lustre: lustre-MDT0000: new disk, initializing [10016.469778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10016.472785] Lustre: Skipped 1 previous similar message [10016.487302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10021.038593] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10032.493111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10032.675336] Lustre: 255637:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10032.754708] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10032.760511] Lustre: Skipped 1 previous similar message [10032.891240] Lustre: lustre-MDT0001: new disk, initializing [10033.156653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10033.175954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10037.866429] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10046.055199] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10049.979258] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10058.452856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10058.994070] Lustre: lustre-OST0000: new disk, initializing [10059.010201] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10059.024593] Lustre: 257381:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10060.531822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10060.554964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10060.703181] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10066.830398] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10078.919994] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10086.272179] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10086.594190] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10092.219660] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10092.535114] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10096.612886] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10096.620218] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10096.647529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10096.663378] Lustre: Skipped 5 previous similar messages [10101.753208] Lustre: server umount lustre-OST0000 complete [10109.922985] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10109.929304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10109.952496] Lustre: Skipped 1 previous similar message [10109.965296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10113.845090] Lustre: server umount lustre-MDT0000 complete [10116.066181] LustreError: 256497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10116.093212] LustreError: 256497:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10118.420160] LustreError: 255629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248603 with bad export cookie 6029349204830762294 [10118.431179] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10118.431229] LustreError: 255629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10118.969265] Lustre: server umount lustre-MDT0001 complete [10147.906252] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [10162.118111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10163.104426] LustreError: 260174:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10163.251372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10163.260239] Lustre: Skipped 2 previous similar messages [10167.502197] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10177.864883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10184.964517] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10190.174472] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10200.951359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10210.137395] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10221.002704] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10221.120404] Lustre: lustre-OST0001: new disk, initializing [10221.125648] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10221.130668] Lustre: 262432:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10222.834200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10222.850274] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10222.989409] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10227.585994] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10239.378511] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10244.059979] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10251.350986] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10251.443483] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10260.455309] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10260.470375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10260.488929] Lustre: Skipped 1 previous similar message [10260.492948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10260.499076] Lustre: Skipped 2 previous similar messages [10265.646613] Lustre: server umount lustre-MDT0000 complete [10269.171383] LustreError: 260175:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10269.200501] LustreError: 260175:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10270.283275] LustreError: 260155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248755 with bad export cookie 6029349204830763176 [10270.288354] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10270.296518] LustreError: 260155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10270.827891] Lustre: server umount lustre-MDT0001 complete [10286.808593] Lustre: server umount lustre-OST0000 complete [10290.673286] Lustre: 238547:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781248759/real 1781248759] req@ffff9f43c76043c0 x1867773940843136/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781248775 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10291.680880] Lustre: 238545:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781248760/real 1781248760] req@ffff9f43c7d14b40 x1867773940843392/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781248776 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10292.247452] Lustre: server umount lustre-OST0001 complete [10316.746161] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [10330.409995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10331.330791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10331.345751] Lustre: Skipped 3 previous similar messages [10336.499511] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10336.759564] LustreError: 265454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10336.781361] LustreError: 265454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10347.685627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10348.003541] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10348.015744] Lustre: Skipped 4 previous similar messages [10353.602893] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10357.351319] Lustre: 266564:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10365.589460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10372.781633] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10377.291616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10383.265658] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10389.006266] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10390.381138] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10399.668292] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10403.830995] Lustre: 268405:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10410.000421] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10410.104869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10421.798941] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 03:21:44 (1781248904) [10440.176103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10440.186623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10440.187735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10440.193469] Lustre: Skipped 5 previous similar messages [10445.164487] Lustre: server umount lustre-MDT0000 complete [10449.717988] LustreError: 265435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248934 with bad export cookie 6029349204830769952 [10449.722265] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10449.741459] LustreError: 265435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10464.224815] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10464.663270] Lustre: server umount lustre-MDT0001 complete [10474.614326] Lustre: server umount lustre-OST0000 complete [10478.896227] Lustre: server umount lustre-OST0001 complete [10485.588816] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [10497.320837] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [10546.323278] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [10555.297940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10555.624961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10555.699593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10555.794456] Lustre: lustre-MDT0000: new disk, initializing [10555.887436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10560.134100] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10570.017688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10570.186904] Lustre: 272600:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10570.503814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10574.536353] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10587.654449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10587.879714] Lustre: 274198:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10587.987153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10587.991992] Lustre: Skipped 5 previous similar messages [10590.017605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10593.419527] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10603.543566] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10603.634290] Lustre: 275052:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10605.558170] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10605.569908] Lustre: Skipped 1 previous similar message [10605.674907] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10610.021462] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10619.692302] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10623.355742] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10623.359853] Lustre: Skipped 1 previous similar message [10637.410135] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 03:25:20 (1781249120) [10641.385322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10641.395979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10641.415503] Lustre: Skipped 7 previous similar messages [10641.426582] Lustre: Skipped 12 previous similar messages [10641.819179] Lustre: server umount lustre-MDT0000 complete [10645.922195] LustreError: 272592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781249130 with bad export cookie 6029349204830777358 [10645.923716] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10645.929774] LustreError: 272592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10646.435853] Lustre: server umount lustre-MDT0001 complete [10660.685172] Lustre: server umount lustre-OST0000 complete [10662.560751] Lustre: 238546:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781249131/real 1781249131] req@ffff9f43cbd9d680 x1867773940994432/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781249147 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10664.580958] Lustre: server umount lustre-OST0001 complete [10673.346752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10673.912525] LustreError: 277281:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10673.928718] LustreError: 277281:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [10679.380464] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10688.323906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10692.592355] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10697.695456] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10701.024758] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10707.662293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10713.784621] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10719.624904] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10735.394384] loop1: detected capacity change from 0 to 204800000 [10735.638700] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10735.646628] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10735.654089] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10740.782265] loop1: detected capacity change from 0 to 204800000 [10748.333814] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10748.569316] LustreError: 277269:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs21964: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10748.589135] LustreError: 277269:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs21964 [10748.600750] LustreError: 277269:0:(mgs_handler.c:517:mgs_target_reg()) fs21964-MDT0001: Error checking target index: rc = -13 [10748.615883] LustreError: fs21964-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10748.644402] LustreError: 280658:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [10748.659925] LustreError: 280658:0:(tgt_mount.c:2084:server_put_super()) no obd fs21964-MDT0001 [10748.679238] LustreError: 280658:0:(tgt_mount.c:116:server_deregister_mount()) fs21964-MDT0001 not registered [10748.759853] Lustre: server umount fs21964-MDT0001 complete [10748.777548] LustreError: 280658:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10762.351159] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10762.581894] Lustre: Setting parameter fs21964-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs21964-MDT0001 [10762.714748] Lustre: srv-fs21964-MDT0001: No data found on store. Initialize space. [10762.722777] Lustre: Skipped 4 previous similar messages [10762.907146] Lustre: fs21964-MDT0001: new disk, initializing [10762.919183] Lustre: Skipped 3 previous similar messages [10763.097092] Lustre: srv-fs21964-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10763.118429] Lustre: 281125:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21964-MDT0001-mdtlov: can't create new llog: rc = -115 [10763.130788] LustreError: 281125:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21964-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10771.944573] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10781.179249] Lustre: server umount fs21964-MDT0001 complete [10795.637923] loop1: detected capacity change from 0 to 204800000 [10800.611665] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10802.007811] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10802.520445] Lustre: 282678:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21964-MDT0001-mdtlov: can't create new llog: rc = -115 [10802.536134] LustreError: 282678:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21964-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10807.338797] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10812.000437] Lustre: server umount fs21964-MDT0001 complete [10822.632183] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.10@tcp (stopping) [10822.639835] Lustre: Skipped 1 previous similar message [10826.214284] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10826.227443] Lustre: Skipped 3 previous similar messages [10828.434474] Lustre: server umount lustre-MDT0000 complete [10836.603689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10836.738211] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10842.110868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10842.716630] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10850.907578] loop1: detected capacity change from 0 to 204800000 [10857.205890] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10857.540842] Lustre: 285094:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21964-MDT0001-mdtlov: can't create new llog: rc = -115 [10857.550261] LustreError: 285094:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21964-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10862.015664] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10866.941575] Lustre: server umount fs21964-MDT0001 complete [10897.908923] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10897.918080] LustreError: Skipped 1 previous similar message [10899.723530] Lustre: server umount lustre-OST0000 complete [10910.699638] Lustre: server umount lustre-MDT0000 complete [10915.227540] LustreError: 277263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781249400 with bad export cookie 6029349204830780039 [10915.233740] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10915.238772] LustreError: 277263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10915.672558] Lustre: server umount lustre-MDT0001 complete [10928.895546] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [10932.535868] Key type lgssc unregistered [10933.019679] LNet: 287406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10933.028094] LNetError: 287406:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10933.061987] LNet: Removed LNI 192.168.201.110@tcp [10934.069625] Key type .llcrypt unregistered [10934.071894] Key type ._llcrypt unregistered [10958.753762] Key type ._llcrypt registered [10958.763260] Key type .llcrypt registered [10958.901444] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [10960.342132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10960.390939] alg: No test for adler32 (adler32-zlib) [10961.453892] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [10961.660714] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [10963.395879] Key type lgssc registered [10965.460841] Lustre: Echo OBD driver; http://www.lustre.org/ [10974.262562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10974.279379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10975.931262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10980.775140] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10985.449212] LustreError: 288660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10985.467360] LustreError: 288660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10990.567792] LustreError: 288661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10991.288665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10991.863232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10996.901561] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11004.813831] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11010.111774] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11017.989735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11018.436661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11020.472087] LustreError: 290262:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11025.356472] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11025.903412] LustreError: 290388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11031.034385] LustreError: 290263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11032.072269] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11040.190570] LustreError: 290262:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11040.225906] LustreError: 290262:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11056.631210] LustreError: 290264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11056.640619] LustreError: 290264:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [11057.051587] loop1: detected capacity change from 0 to 204800000 [11057.250364] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11057.266443] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11057.284685] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11063.308356] loop1: detected capacity change from 0 to 204800000 [11069.162590] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11069.306520] Lustre: fs916-OST0000: new disk, initializing [11069.309504] Lustre: srv-fs916-OST0000: No data found on store. Initialize space. [11069.314398] Lustre: 292056:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11069.379079] Lustre: fs916-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11075.974817] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11085.642404] Lustre: server umount fs916-OST0000 complete [11091.395160] LustreError: 290264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11091.415833] LustreError: 290264:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [11098.657809] loop1: detected capacity change from 0 to 204800000 [11102.696715] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11105.211879] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11105.465274] Lustre: fs916-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11112.060460] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11118.059931] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11121.640099] loop2: detected capacity change from 0 to 204800000 [11121.997494] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11122.011128] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11122.022035] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11129.239805] loop2: detected capacity change from 0 to 204800000 [11136.232794] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11136.361974] LustreError: fs916-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11136.370671] LustreError: fs916-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11136.377588] LustreError: 295063:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [11136.385427] LustreError: 295063:0:(tgt_mount.c:2084:server_put_super()) no obd fs916-OST0001 [11136.389114] LustreError: 295063:0:(tgt_mount.c:116:server_deregister_mount()) fs916-OST0001 not registered [11136.413919] Lustre: server umount fs916-OST0001 complete [11136.416068] LustreError: 295063:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11149.049972] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11149.250745] Lustre: fs916-OST0001: new disk, initializing [11149.254440] Lustre: srv-fs916-OST0001: No data found on store. Initialize space. [11149.260848] Lustre: 295614:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11149.353483] Lustre: fs916-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11155.878607] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11157.942366] LustreError: 290264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11157.962689] LustreError: 290264:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [11165.344034] Lustre: server umount fs916-OST0000 complete [11176.194682] Lustre: server umount fs916-OST0001 complete [11190.567802] loop1: detected capacity change from 0 to 204800000 [11197.403176] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11197.613076] Lustre: fs916-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11204.434940] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11212.129796] loop2: detected capacity change from 0 to 204800000 [11218.336719] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11225.049765] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11235.236547] Lustre: server umount fs916-OST0000 complete [11245.256478] Lustre: server umount fs916-OST0001 complete [11256.305043] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11261.409801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11261.432674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11261.433576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11261.448944] Lustre: Skipped 1 previous similar message [11264.752815] Lustre: server umount lustre-OST0000 complete [11271.661241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11271.681288] Lustre: Skipped 1 previous similar message [11271.689043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11271.707273] Lustre: Skipped 1 previous similar message [11275.518608] Lustre: server umount lustre-MDT0000 complete [11279.930588] LustreError: 292784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781249764 with bad export cookie 3014553272261539049 [11279.953846] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11279.957613] LustreError: 292784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11280.311746] Lustre: server umount lustre-MDT0001 complete [11293.827034] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [11298.227270] Key type lgssc unregistered [11298.643512] LNet: 301472:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11298.648688] LNetError: 301472:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11299.687257] LNet: Removed LNI 192.168.201.110@tcp [11300.738723] Key type .llcrypt unregistered [11300.744471] Key type ._llcrypt unregistered [11337.045336] Key type ._llcrypt registered [11337.047874] Key type .llcrypt registered [11337.167195] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [11356.698607] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [11357.749906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11357.817784] alg: No test for adler32 (adler32-zlib) [11358.945651] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [11359.299103] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [11360.970507] Key type lgssc registered [11362.401797] Lustre: Echo OBD driver; http://www.lustre.org/ [11402.372708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11402.404754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11403.771099] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11403.811360] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11403.948454] Lustre: lustre-MDT0000: new disk, initializing [11404.086125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11404.113922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11409.207638] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11419.843120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11419.965825] Lustre: 305300:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11420.037401] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11420.045878] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11420.047470] Lustre: Skipped 1 previous similar message [11420.184462] Lustre: lustre-MDT0001: new disk, initializing [11420.304286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11420.355562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11420.363899] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11424.388942] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11432.419719] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11436.705376] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11444.463136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11444.855689] Lustre: lustre-OST0000: new disk, initializing [11444.860765] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11444.867054] Lustre: 307042:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11444.941723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11450.927621] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11450.957308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11450.969212] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11451.119668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11458.273452] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11460.632568] Lustre: Mounted lustre-client [11461.756159] Lustre: Unmounted lustre-client [11470.819473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11470.847644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11475.945801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11475.963208] Lustre: Skipped 2 previous similar messages [11476.428541] Lustre: server umount lustre-OST0000 complete [11481.059027] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11481.067845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11481.085414] Lustre: Skipped 2 previous similar messages [11481.100230] Lustre: Skipped 1 previous similar message [11486.186683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11486.197099] Lustre: Skipped 1 previous similar message [11494.368190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11494.543728] Lustre: server umount lustre-MDT0000 complete [11497.645710] LustreError: 305292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781249982 with bad export cookie 17451878695938126766 [11497.647692] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11497.661796] LustreError: 305292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11497.983115] Lustre: server umount lustre-MDT0001 complete [11507.053198] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [11509.392424] Key type lgssc unregistered [11516.833166] LNet: 309076:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11516.845228] LNetError: 309076:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11593.714235] LNet: Removed LNI 192.168.201.110@tcp [11594.501195] Key type .llcrypt unregistered [11594.504404] Key type ._llcrypt unregistered [11608.518600] Key type ._llcrypt registered [11608.520646] Key type .llcrypt registered [11608.613830] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [11620.649448] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [11621.654729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11621.671326] alg: No test for adler32 (adler32-zlib) [11622.665298] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [11622.843352] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [11624.496173] Key type lgssc registered [11625.436897] Lustre: Echo OBD driver; http://www.lustre.org/ [11652.105543] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 03:42:15 (1781250135) [11657.798675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11657.833950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11659.043874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11659.071411] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11659.140794] Lustre: lustre-MDT0000: new disk, initializing [11659.229528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11659.248633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11662.730055] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11670.874484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11670.983811] Lustre: 312244:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11671.034151] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11671.037448] Lustre: Skipped 1 previous similar message [11671.096706] Lustre: lustre-MDT0001: new disk, initializing [11671.206855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11671.244719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11671.251746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11674.670537] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11681.933104] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11685.089650] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11690.847558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11691.038381] Lustre: lustre-OST0000: new disk, initializing [11691.041455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11691.046635] Lustre: 313986:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11691.129072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11695.763425] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11697.290159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11697.309061] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11697.452543] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11702.357639] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11711.959561] Lustre: server umount lustre-OST0000 complete [11717.601265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11717.605348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11717.623499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11718.114866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11718.137504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11718.144184] Lustre: Skipped 1 previous similar message [11723.236163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11723.239451] Lustre: Skipped 1 previous similar message [11728.352280] Lustre: 310419:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781250197/real 1781250197] req@000000008fc41d81 x1867776112228608/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781250213 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11728.357251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11728.382228] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11728.398369] Lustre: Skipped 1 previous similar message [11729.376166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11729.646093] Lustre: server umount lustre-MDT0000 complete [11732.838922] LustreError: 312235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781250217 with bad export cookie 2604884237832570628 [11732.840196] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11732.851292] LustreError: 312235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11733.092932] Lustre: server umount lustre-MDT0001 complete [11741.640912] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [11743.839411] Key type lgssc unregistered [11744.070870] LNet: 315616:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11744.075701] LNetError: 315616:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11744.086294] LNet: Removed LNI 192.168.201.110@tcp [11744.681749] Key type .llcrypt unregistered [11744.684759] Key type ._llcrypt unregistered [11764.418898] Key type ._llcrypt registered [11764.420929] Key type .llcrypt registered [11764.483907] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [11765.120787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11765.252332] alg: No test for adler32 (adler32-zlib) [11766.259498] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [11766.460655] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [11768.176199] Key type lgssc registered [11768.999697] Lustre: Echo OBD driver; http://www.lustre.org/ [11775.048950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11775.063529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11776.450105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11780.105429] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11786.723655] LustreError: 317001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11786.736054] LustreError: 317001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11787.372174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11787.778747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11791.942302] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11796.240912] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11799.011535] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11804.152071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11804.402243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11808.759700] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11812.859484] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11830.021656] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11831.375366] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11838.434314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11838.438839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11838.446760] Lustre: Skipped 1 previous similar message [11838.456208] Lustre: Skipped 2 previous similar messages [11839.965723] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.10@tcp (stopping) [11841.365445] Lustre: server umount lustre-MDT0000 complete [11843.565726] LustreError: 316997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11843.578320] LustreError: 316997:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11844.724809] LustreError: 316980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781250329 with bad export cookie 4205102763764673264 [11844.725606] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11844.732777] LustreError: 316980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11845.051949] LustreError: 318443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11845.078508] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.10@tcp (stopping) [11848.680700] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11848.685359] LustreError: 318443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11848.701926] Lustre: Skipped 1 previous similar message [11848.704295] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11851.092458] Lustre: server umount lustre-MDT0001 complete [11856.852628] Lustre: server umount lustre-OST0000 complete [11864.439386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11867.441216] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11869.776537] Lustre: 320747:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.110@tcp [11871.364305] Lustre: server umount MGS complete [11879.149911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11879.593330] LustreError: 321318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11879.601659] LustreError: 321318:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11879.650832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11883.269616] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11889.132722] LustreError: 321319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11889.140892] LustreError: 321319:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11890.350524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11890.579341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11894.322920] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11902.817441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11903.225251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11908.109373] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11912.712498] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11913.991920] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11916.895512] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11918.238294] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11923.942963] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11923.949086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11923.962773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11923.965143] Lustre: Skipped 2 previous similar messages [11925.988979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11925.995937] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11927.787615] Lustre: server umount lustre-OST0000 complete [11931.617878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11931.628293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11934.181523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11934.188956] Lustre: Skipped 4 previous similar messages [11936.948104] Lustre: server umount lustre-MDT0000 complete [11939.308919] LustreError: 323374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11939.320913] LustreError: 323374:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11940.108529] LustreError: 322037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781250425 with bad export cookie 4205102763764674104 [11940.111346] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11940.117972] LustreError: 322037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11940.507643] Lustre: server umount lustre-MDT0001 complete [11949.632583] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [11952.195492] Key type lgssc unregistered [11952.481982] LNet: 324494:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11952.486265] LNetError: 324494:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11952.500971] LNet: Removed LNI 192.168.201.110@tcp [11953.277139] Key type .llcrypt unregistered [11953.283231] Key type ._llcrypt unregistered [11969.315508] Key type ._llcrypt registered [11969.317929] Key type .llcrypt registered [11969.419293] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [11983.182570] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [11984.666503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11984.677219] alg: No test for adler32 (adler32-zlib) [11985.755208] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [11985.929787] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [11987.609443] Key type lgssc registered [11988.463414] Lustre: Echo OBD driver; http://www.lustre.org/ [12018.038716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12018.052460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12019.303270] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12019.332428] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12019.453281] Lustre: lustre-MDT0000: new disk, initializing [12019.552911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12019.577447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12022.964886] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12033.000498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12033.106227] Lustre: 327615:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12033.154512] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12033.164814] Lustre: Skipped 1 previous similar message [12033.265296] Lustre: lustre-MDT0001: new disk, initializing [12033.362644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12033.432194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12033.452194] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12037.070622] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12043.552877] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12046.602582] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12052.054945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12052.240894] Lustre: lustre-OST0000: new disk, initializing [12052.243923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12052.250712] Lustre: 329355:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12052.324737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12056.776809] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12060.223307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12060.230826] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12060.348729] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12062.709305] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12066.332802] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12066.523854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12069.864341] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12070.005419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12076.005895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12076.009244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12076.014305] LustreError: Skipped 1 previous similar message [12076.015290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12076.041308] Lustre: Skipped 1 previous similar message [12077.864902] Lustre: server umount lustre-OST0000 complete [12084.194402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12084.196416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12084.200381] Lustre: Skipped 1 previous similar message [12084.214439] Lustre: Skipped 2 previous similar messages [12087.165714] Lustre: server umount lustre-MDT0000 complete [12090.243703] LustreError: 327604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781250575 with bad export cookie 6643572519219712204 [12090.244861] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12090.257835] LustreError: 327604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12090.567740] Lustre: server umount lustre-MDT0001 complete [12098.158937] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 03:49:41 (1781250581) [12111.847221] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [12120.172880] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [12149.367677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12149.596975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12149.613489] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12149.671637] Lustre: lustre-MDT0000: new disk, initializing [12149.755778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12149.769613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12153.190498] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12161.341992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12161.424361] Lustre: 333928:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12161.456842] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12161.459875] Lustre: Skipped 1 previous similar message [12161.513984] Lustre: lustre-MDT0001: new disk, initializing [12161.588356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12161.615966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12161.621696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12164.766233] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12170.580371] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12173.654710] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12178.629310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12178.826728] Lustre: lustre-OST0000: new disk, initializing [12178.829883] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12178.834948] Lustre: 335671:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12178.909959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12183.050473] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12184.620836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12184.630942] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12184.723848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12188.859592] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12210.148433] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12210.150537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12210.157158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12210.175160] Lustre: Skipped 1 previous similar message [12215.272453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12215.277951] Lustre: Skipped 2 previous similar messages [12215.504977] Lustre: server umount lustre-OST0000 complete [12218.141390] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12218.150426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12220.387492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12220.397856] Lustre: Skipped 2 previous similar messages [12224.368298] Lustre: server umount lustre-MDT0000 complete [12225.505874] LustreError: 335561:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12225.526616] LustreError: 335561:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12226.913814] LustreError: 333922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781250711 with bad export cookie 6643572519219713086 [12226.916248] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12226.924173] LustreError: 333922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12227.170582] Lustre: server umount lustre-MDT0001 complete [12234.962849] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12237.334237] Key type lgssc unregistered [12237.583528] LNet: 337689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12237.595862] LNetError: 337689:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12237.611485] LNet: Removed LNI 192.168.201.110@tcp [12238.392102] Key type .llcrypt unregistered [12238.395143] Key type ._llcrypt unregistered [12303.768516] Key type ._llcrypt registered [12303.770955] Key type .llcrypt registered [12303.839833] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12304.068710] Key type .llcrypt unregistered [12304.072409] Key type ._llcrypt unregistered [12362.122480] Key type ._llcrypt registered [12362.127477] Key type .llcrypt registered [12362.189092] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12362.366602] Key type .llcrypt unregistered [12362.368710] Key type ._llcrypt unregistered [12383.034375] Key type ._llcrypt registered [12383.038229] Key type .llcrypt registered [12383.107869] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12383.276308] Key type .llcrypt unregistered [12383.278197] Key type ._llcrypt unregistered [12403.763194] Key type ._llcrypt registered [12403.765403] Key type .llcrypt registered [12403.832317] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12404.033565] Key type .llcrypt unregistered [12404.035967] Key type ._llcrypt unregistered [12465.453154] Key type ._llcrypt registered [12465.455573] Key type .llcrypt registered [12465.581974] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12465.870714] Key type .llcrypt unregistered [12465.875606] Key type ._llcrypt unregistered [12489.012978] Key type ._llcrypt registered [12489.014639] Key type .llcrypt registered [12489.117891] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12489.408379] Key type .llcrypt unregistered [12489.410563] Key type ._llcrypt unregistered [12511.394261] Key type ._llcrypt registered [12511.395734] Key type .llcrypt registered [12511.472972] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12511.688252] Key type .llcrypt unregistered [12511.692618] Key type ._llcrypt unregistered [12533.704182] Key type ._llcrypt registered [12533.705428] Key type .llcrypt registered [12533.759610] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12533.962573] Key type .llcrypt unregistered [12533.964311] Key type ._llcrypt unregistered [12552.410652] Key type ._llcrypt registered [12552.411924] Key type .llcrypt registered [12552.456634] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12552.593372] Key type .llcrypt unregistered [12552.594685] Key type ._llcrypt unregistered [12571.684673] Key type ._llcrypt registered [12571.686293] Key type .llcrypt registered [12571.762256] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12571.955418] Key type .llcrypt unregistered [12571.956805] Key type ._llcrypt unregistered [12612.554147] Key type ._llcrypt registered [12612.555343] Key type .llcrypt registered [12612.628238] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12612.804595] Key type .llcrypt unregistered [12612.806457] Key type ._llcrypt unregistered [12634.465627] Key type ._llcrypt registered [12634.467425] Key type .llcrypt registered [12634.541682] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12634.711877] Key type .llcrypt unregistered [12634.713564] Key type ._llcrypt unregistered [12656.628785] Key type ._llcrypt registered [12656.631383] Key type .llcrypt registered [12656.700452] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12656.880262] Key type .llcrypt unregistered [12656.881328] Key type ._llcrypt unregistered [12691.716379] Key type ._llcrypt registered [12691.719481] Key type .llcrypt registered [12691.805994] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [12702.941580] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [12703.668250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12703.679538] alg: No test for adler32 (adler32-zlib) [12704.595931] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [12704.733582] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [12706.376279] Key type lgssc registered [12707.227455] Lustre: Echo OBD driver; http://www.lustre.org/ [12729.053366] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12729.071809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12730.227775] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12730.240587] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12730.280474] Lustre: tst250fs-MDT0000: new disk, initializing [12730.316629] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12730.324598] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12732.548150] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12738.668123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12738.719338] Lustre: 357427:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12738.746299] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12738.748923] Lustre: Skipped 1 previous similar message [12738.789353] Lustre: tst250fs-MDT0001: new disk, initializing [12738.851491] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12738.884242] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12738.896349] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12740.954715] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12743.934730] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12747.760396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12747.897647] Lustre: tst250fs-OST0000: new disk, initializing [12747.900307] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12747.903310] Lustre: 359023:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12747.962023] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12750.053899] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12750.062287] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12750.135833] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12751.210485] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12758.138391] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12758.262885] Lustre: tst250fs-OST0001: new disk, initializing [12758.265913] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12758.270666] Lustre: 359878:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12758.346050] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12759.019905] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12759.030545] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12759.075160] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12761.864753] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12765.616412] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12780.000963] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12780.004301] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12780.011978] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12784.614044] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12784.617959] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12784.631432] Lustre: Skipped 3 previous similar messages [12789.730953] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12789.735650] Lustre: Skipped 3 previous similar messages [12794.336362] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12794.422692] Lustre: server umount tst250fs-MDT0000 complete [12796.305909] LustreError: 358205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781251281 with bad export cookie 5948952692298707742 [12796.312571] LustreError: 358205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12796.314166] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12796.487710] Lustre: server umount tst250fs-MDT0001 complete [12808.211445] Lustre: server umount tst250fs-OST0000 complete [12820.887913] Lustre: server umount tst250fs-OST0001 complete [12833.160365] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [12838.486268] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [12857.399198] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 04:02:21 (1781251341) [12858.423826] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12867.925590] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [12874.414967] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [12892.125974] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12597 sec ========= 04:02:56 (1781251376) [12892.999890] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:02:57 (1781251377) === [12894.746425] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:02:58 (1781251378) === [12914.533924] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12915.947165] Key type lgssc unregistered [12916.105688] LNet: 368902:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12916.110758] LNetError: 368902:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12916.125189] LNet: Removed LNI 192.168.201.110@tcp [12916.611150] Key type .llcrypt unregistered [12916.612878] Key type ._llcrypt unregistered