[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1462771298 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002016] Switched APIC routing to physical x2apic. [ 0.004025] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009000] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009000] pid_max: default: 32768 minimum: 301 [ 0.009000] LSM: Security Framework initializing [ 0.009000] Yama: becoming mindful. [ 0.009000] SELinux: Initializing. [ 0.009000] *** VALIDATE selinux *** [ 0.009000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.009000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.009000] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.009000] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.009000] *** VALIDATE tmpfs *** [ 0.009000] *** VALIDATE proc *** [ 0.009000] *** VALIDATE cgroup *** [ 0.009000] *** VALIDATE cgroup2 *** [ 0.009000] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.009000] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.009000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.009000] Spectre V2 : User space: Vulnerable [ 0.009000] Speculative Store Bypass: Vulnerable [ 0.009000] debug: unmapping init [mem 0xffffffff9ee59000-0xffffffff9ee60fff] [ 0.009000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.009833] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.010039] ... version: 2 [ 0.011014] ... bit width: 48 [ 0.012011] ... generic registers: 4 [ 0.013014] ... value mask: 0000ffffffffffff [ 0.014014] ... max period: 00007fffffffffff [ 0.015013] ... fixed-purpose events: 3 [ 0.016012] ... event mask: 000000070000000f [ 0.018268] rcu: Hierarchical SRCU implementation. [ 0.020583] smp: Bringing up secondary CPUs ... [ 0.021634] x86: Booting SMP configuration: [ 0.022027] .... node #0, CPUs: #1 #2 #3 [ 0.169171] smp: Brought up 1 node, 4 CPUs [ 0.171022] smpboot: Max logical packages: 1 [ 0.172021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.203561] node 0 deferred pages initialised in 30ms [ 0.208528] devtmpfs: initialized [ 0.209354] x86/mm: Memory block size: 128MB [ 0.211863] gcov: version magic: 0x41383552 [ 0.215341] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.219251] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.223369] pinctrl core: initialized pinctrl subsystem [ 0.226211] [ 0.226698] ************************************************************* [ 0.229019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.231018] ** ** [ 0.235022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.238015] ** ** [ 0.240034] ** This means that this kernel is built to expose internal ** [ 0.243020] ** IOMMU data structures, which may compromise security on ** [ 0.246017] ** your system. ** [ 0.248019] ** ** [ 0.251018] ** If you see this message and you are not debugging the ** [ 0.253066] ** kernel, report this immediately to your vendor! ** [ 0.256048] ** ** [ 0.258020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.261017] ************************************************************* [ 0.264934] NET: Registered protocol family 16 [ 0.266639] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.270074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.273069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.277131] cpuidle: using governor menu [ 0.278844] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.281732] PCI: Using configuration type 1 for base access [ 0.284139] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.291158] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.292024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.296234] cryptd: max_cpu_qlen set to 1000 [ 0.298271] ACPI: Added _OSI(Module Device) [ 0.299014] ACPI: Added _OSI(Processor Device) [ 0.301010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.303017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.308685] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.317014] ACPI: Interpreter enabled [ 0.319063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.320014] ACPI: Using IOAPIC for interrupt routing [ 0.322140] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.326512] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.342240] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.345048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.348023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.353085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.359829] acpiphp: Slot [2] registered [ 0.362179] acpiphp: Slot [5] registered [ 0.364128] acpiphp: Slot [6] registered [ 0.365310] acpiphp: Slot [7] registered [ 0.367303] acpiphp: Slot [8] registered [ 0.369159] acpiphp: Slot [9] registered [ 0.371178] acpiphp: Slot [10] registered [ 0.373161] acpiphp: Slot [3] registered [ 0.375114] acpiphp: Slot [4] registered [ 0.376202] acpiphp: Slot [11] registered [ 0.378140] acpiphp: Slot [12] registered [ 0.380186] acpiphp: Slot [13] registered [ 0.382164] acpiphp: Slot [14] registered [ 0.384196] acpiphp: Slot [15] registered [ 0.386122] acpiphp: Slot [16] registered [ 0.388157] acpiphp: Slot [17] registered [ 0.390110] acpiphp: Slot [18] registered [ 0.392111] acpiphp: Slot [19] registered [ 0.393166] acpiphp: Slot [20] registered [ 0.395103] acpiphp: Slot [21] registered [ 0.397145] acpiphp: Slot [22] registered [ 0.399116] acpiphp: Slot [23] registered [ 0.400158] acpiphp: Slot [24] registered [ 0.402111] acpiphp: Slot [25] registered [ 0.404138] acpiphp: Slot [26] registered [ 0.405354] acpiphp: Slot [27] registered [ 0.408164] acpiphp: Slot [28] registered [ 0.410126] acpiphp: Slot [29] registered [ 0.412126] acpiphp: Slot [30] registered [ 0.414105] acpiphp: Slot [31] registered [ 0.416103] PCI host bridge to bus 0000:00 [ 0.417167] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.420029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.424027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.427027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.430024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.433031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.435301] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.438207] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.442658] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.454851] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.459000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.463021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.465018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.470021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.473609] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.476934] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.479048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.482941] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.489015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.502015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.507964] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.515606] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.522025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.529022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.556020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.564715] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.571021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.579020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.593021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.603182] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.610018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.621016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.635016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.648233] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.769053] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.779000] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.815026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.833929] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.850023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.860027] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.883022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.899505] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.912021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.929023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.966000] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.980000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.982378] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.984384] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.987404] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.989236] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.994589] iommu: Default domain type: Passthrough [ 0.996653] SCSI subsystem initialized [ 0.997618] ACPI: bus type USB registered [ 0.998247] usbcore: registered new interface driver usbfs [ 1.001111] usbcore: registered new interface driver hub [ 1.003091] usbcore: registered new device driver usb [ 1.005195] pps_core: LinuxPPS API ver. 1 registered [ 1.007012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.011062] PTP clock support registered [ 1.014165] EDAC MC: Ver: 3.0.0 [ 1.015211] PCI: Using ACPI for IRQ routing [ 1.017047] NetLabel: Initializing [ 1.018010] NetLabel: domain hash size = 128 [ 1.020014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.022101] NetLabel: unlabeled traffic allowed by default [ 1.024283] vgaarb: loaded [ 1.025375] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.027016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.032187] clocksource: Switched to clocksource kvm-clock [ 1.146970] VFS: Disk quotas dquot_6.6.0 [ 1.148677] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.151333] *** VALIDATE ramfs *** [ 1.152671] *** VALIDATE hugetlbfs *** [ 1.154246] pnp: PnP ACPI init [ 1.156669] pnp: PnP ACPI: found 6 devices [ 1.172255] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.175691] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.178224] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.180610] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.183047] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.185430] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.188381] NET: Registered protocol family 2 [ 1.191453] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.197142] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.201215] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.206746] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.210537] TCP: Hash tables configured (established 65536 bind 65536) [ 1.215250] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.218596] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.221823] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.225293] NET: Registered protocol family 1 [ 1.227832] RPC: Registered named UNIX socket transport module. [ 1.230504] RPC: Registered udp transport module. [ 1.231832] RPC: Registered tcp transport module. [ 1.234104] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.236818] NET: Registered protocol family 44 [ 1.238670] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.241436] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.245097] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.248641] PCI: CLS 0 bytes, default 64 [ 1.252758] Unpacking initramfs... [ 4.499121] debug: unmapping init [mem 0xffff9bd2fcc54000-0xffff9bd2fffbffff] [ 4.504582] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.508191] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.511564] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.646111] Initialise system trusted keyrings [ 5.647469] Key type blacklist registered [ 5.650307] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.671665] zbud: loaded [ 5.685099] *** VALIDATE nfs *** [ 5.687584] *** VALIDATE nfs4 *** [ 5.690615] pstore: using deflate compression [ 5.697183] Platform Keyring initialized [ 6.039282] NET: Registered protocol family 38 [ 6.041164] Key type asymmetric registered [ 6.042722] Asymmetric key parser 'x509' registered [ 6.044911] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.048153] io scheduler mq-deadline registered [ 6.049866] io scheduler kyber registered [ 6.051716] io scheduler bfq registered [ 6.157490] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.161370] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.165202] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.169792] ACPI: Power Button [PWRF] [ 6.180563] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.198558] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.214061] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.222550] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.240327] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.273751] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.312100] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.318548] Non-volatile memory driver v1.3 [ 6.320319] Linux agpgart interface v0.103 [ 6.348715] virtio_blk virtio1: [vda] 149976 512-byte logical blocks (76.8 MB/73.2 MiB) [ 6.352996] vda: detected capacity change from 0 to 76787712 [ 6.631674] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.636164] vdb: detected capacity change from 0 to 1073741824 [ 6.666949] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.669943] vdc: detected capacity change from 0 to 2621440000 [ 6.690461] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.777400] vdd: detected capacity change from 0 to 2621440000 [ 6.794589] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.797712] vde: detected capacity change from 0 to 4294967296 [ 6.822397] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.829801] vdf: detected capacity change from 0 to 4294967296 [ 6.843804] libphy: Fixed MDIO Bus: probed [ 6.855208] usbcore: registered new interface driver usbserial_generic [ 6.857698] usbserial: USB Serial support registered for generic [ 6.860126] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.864766] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.866628] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.869481] mousedev: PS/2 mouse device common for all mice [ 6.872407] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.878398] rtc_cmos 00:05: RTC can wake from S4 [ 6.880599] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.882460] rtc_cmos 00:05: registered as rtc0 [ 6.882517] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.882572] intel_pstate: CPU model not supported [ 6.893774] hid: raw HID events driver (C) Jiri Kosina [ 6.900696] usbcore: registered new interface driver usbhid [ 6.906220] usbhid: USB HID core driver [ 6.906785] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.911051] drop_monitor: Initializing network drop monitor service [ 6.911206] Initializing XFRM netlink socket [ 6.912744] NET: Registered protocol family 10 [ 6.927095] Segment Routing with IPv6 [ 6.928610] NET: Registered protocol family 17 [ 6.930851] mpls_gso: MPLS GSO support [ 6.935896] RAS: Correctable Errors collector initialized. [ 6.938151] AVX version of gcm_enc/dec engaged. [ 6.940121] AES CTR mode by8 optimization enabled [ 7.042936] sched_clock: Marking stable (7042853458, 0)->(8830766140, -1787912682) [ 7.046908] registered taskstats version 1 [ 7.048950] Loading compiled-in X.509 certificates [ 7.051183] zswap: loaded using pool lzo/zbud [ 7.134688] Key type big_key registered [ 7.157889] Key type encrypted registered [ 7.160213] ima: No TPM chip found, activating TPM-bypass! [ 7.162360] ima: Allocated hash algorithm: sha1 [ 7.165906] ima: No architecture policies found [ 7.167711] evm: Initialising EVM extended attributes: [ 7.170199] evm: security.selinux [ 7.171589] evm: security.ima [ 7.172746] evm: security.capability [ 7.174486] evm: HMAC attrs: 0x1 [ 7.176874] rtc_cmos 00:05: setting system clock to 2026-09-17 22:11:23 UTC (1789683083) [ 7.188705] debug: unmapping init [mem 0xffffffff9fe03000-0xffffffff9fffffff] [ 7.194697] debug: unmapping init [mem 0xffffffff9eb82000-0xffffffff9ee58fff] [ 7.212199] Write protecting the kernel read-only data: 28672k [ 7.216222] debug: unmapping init [mem 0xffffffff9d203000-0xffffffff9d3fffff] [ 7.219497] debug: unmapping init [mem 0xffffffff9db14000-0xffffffff9dbfffff] [ 7.281316] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.293740] systemd[1]: Detected virtualization kvm. [ 7.296883] systemd[1]: Detected architecture x86-64. [ 7.302479] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.330564] systemd[1]: No hostname configured. [ 7.332541] systemd[1]: Set hostname to . [ 7.335571] random: systemd: uninitialized urandom read (16 bytes read) [ 7.340454] systemd[1]: Initializing machine ID from random generator. [ 7.528224] random: ln: uninitialized urandom read (6 bytes read) [ 7.822777] random: systemd: uninitialized urandom read (16 bytes read) [ 7.828066] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 7.844663] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 7.858629] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.162484] device-mapper: uevent: version 1.0.3 [ 10.164979] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 12.502512] random: fast init done [ 12.522961] virtio_net virtio0 ens2: renamed from eth0 [ 13.062014] scsi host0: ata_piix [ 13.284436] scsi host1: ata_piix [ 13.291409] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 13.305064] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 18.071940] random: crng init done [ 18.073934] random: 7 urandom warning(s) missed due to ratelimiting [ 22.001817] dracut-initqueue[568]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 24.917695] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 29.055581] printk: systemd: 26 output lines suppressed due to ratelimiting [ 30.319431] SELinux: Disabled at runtime. [ 30.447723] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 30.461536] systemd[1]: Detected virtualization kvm. [ 30.464195] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 30.890019] hrtimer: interrupt took 11005490 ns [ 33.427809] systemd[1]: initrd-switch-root.service: Succeeded. [ 33.446921] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 33.525035] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 33.547176] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 33.568446] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 33.608426] systemd[1]: Starting Journal Service... Starting Journal Service... [ 33.632653] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd Root File System. [ 34.239649] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 36.238046] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 37.857279] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 38.036451] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 38.711684] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 38.905178] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit)[ 47.657793] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (14s / no limit) [*** ] A start job is running for Configur…only root support (15s / no limit) [** ] A start job is running for Configur…only root support (15s / no limit)[ 49.198139] NFS: Registering the id_resolver key type [ 49.208067] Key type id_resolver registered [ 49.210040] Key type id_legacy registered [* ] A start job is running for Configur…only root support (16s / no limit) [** ] A start job is running for Configur…only root support (16s / no limit) [*** ] A start job is running for Configur…only root support (17s / no limit) [ *** ] A start job is running for Configur…only root support (17s / no limit) [ *** ] A start job is running for Configur…only root support (18s / no limit) [ ***] A start job is running for Configur…only root support (18s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg212-server login: [ 118.738929] libcfs: loading out-of-tree module taints kernel. [ 118.815315] Key type ._llcrypt registered [ 118.817574] Key type .llcrypt registered [ 118.940290] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 136.092163] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 137.684574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 137.698513] alg: No test for adler32 (adler32-zlib) [ 139.144727] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 140.391689] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 142.175237] Key type lgssc registered [ 143.767767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 160.615473] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 216.627204] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 229.658871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 229.730683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 231.184557] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 231.240616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 231.379747] Lustre: lustre-MDT0000: new disk, initializing [ 231.491283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 231.510408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 236.164240] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.446739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 249.547783] Lustre: 6500:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 249.562890] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 249.565569] Lustre: Skipped 1 previous similar message [ 249.680220] Lustre: lustre-MDT0001: new disk, initializing [ 249.758873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 249.824510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 249.843858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 254.150932] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 259.250908] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 269.146180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 269.443554] Lustre: lustre-OST0000: new disk, initializing [ 269.448744] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 269.457380] Lustre: 8437:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 269.547872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 276.729812] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 279.063216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 279.070499] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 279.141571] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 291.663603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 291.838446] Lustre: lustre-OST0001: new disk, initializing [ 291.844630] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 291.851624] Lustre: 9509:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 291.902153] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 298.061230] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 301.633101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 301.643333] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 301.706497] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 310.050344] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 315.860043] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 324.974351] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing check_logdir /tmp/testlogs/ [ 331.722720] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing yml_node [ 336.910095] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 340.052791] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 342.754412] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 344.477799] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 17 18:16:59 EDT 2026 [ 363.794776] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 365.478708] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 382.953974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 382.971093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 382.996061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 383.971062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 383.972698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 388.646381] Lustre: server umount lustre-MDT0000 complete [ 392.586877] LustreError: 6493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789683468 with bad export cookie 1244327118706903357 [ 392.613838] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 394.208106] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 394.208271] LustreError: 10085:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 394.208619] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 394.208624] Lustre: Skipped 3 previous similar messages [ 394.230373] Lustre: Skipped 3 previous similar messages [ 394.249986] LustreError: 10085:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 397.216215] LustreError: 10085:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 397.219838] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 397.257555] Lustre: Skipped 2 previous similar messages [ 399.065269] Lustre: server umount lustre-MDT0001 complete [ 403.423654] Lustre: server umount lustre-OST0000 complete [ 407.378430] Lustre: server umount lustre-OST0001 complete [ 414.396239] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 425.188844] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 462.540427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 462.778547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 462.799492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 462.863175] Lustre: lustre-MDT0000: new disk, initializing [ 462.913479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 462.924590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 467.700638] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 479.066731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 479.223734] Lustre: 15068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 479.303513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 479.312941] Lustre: Skipped 1 previous similar message [ 479.422085] Lustre: lustre-MDT0001: new disk, initializing [ 479.525973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 479.592264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 479.609641] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 484.620264] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 497.384736] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 505.988594] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 514.685452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 515.073865] Lustre: lustre-OST0000: new disk, initializing [ 515.077323] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 515.082840] Lustre: 17037:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 515.165957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 517.041891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 517.072049] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 517.293397] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 524.949955] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 536.537458] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 541.473806] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 541.739840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 546.717491] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 547.010626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 552.929459] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 552.937193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 552.939920] LustreError: Skipped 1 previous similar message [ 552.943659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 552.968326] Lustre: Skipped 1 previous similar message [ 555.371836] Lustre: server umount lustre-OST0000 complete [ 561.645163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 561.658265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 561.676846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 561.679361] Lustre: Skipped 1 previous similar message [ 565.457794] Lustre: server umount lustre-MDT0000 complete [ 568.294411] LustreError: 15078:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.310608] LustreError: 15078:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 570.428326] LustreError: 17036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789683646 with bad export cookie 1244327118706904890 [ 570.437901] LustreError: 17036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 570.449439] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 570.920298] Lustre: server umount lustre-MDT0001 complete [ 579.118719] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 18:20:53 (1789683653) [ 601.695408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 602.518234] LustreError: 19913:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 602.624601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 607.715352] LustreError: 19914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 608.540272] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 616.943721] LustreError: 19914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 616.981039] LustreError: 19914:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 618.324258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 623.915908] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 632.029442] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 638.323674] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 640.606549] Lustre: Failing over lustre-MDT0000 [ 640.904798] Lustre: server umount lustre-MDT0000 complete [ 643.685115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 643.691455] Lustre: Skipped 1 previous similar message [ 643.694826] LustreError: 19908:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 643.708764] LustreError: 19908:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 660.447369] Lustre: 3632:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789683720/real 1789683720] req@ffff9bd37c824700 x1876618868063872/t0(0) o400->MGC192.168.202.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789683736 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 660.464448] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 660.800327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 669.743724] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1144bcebf6e914c0 [ 669.762756] Lustre: MGC192.168.202.112@tcp: Connection restored to 0@lo (at 0@lo) [ 669.927693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 669.934191] Lustre: Skipped 1 previous similar message [ 669.987604] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 673.803371] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 675.298394] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 675.305706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 675.336472] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 684.533703] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 689.509650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 689.518674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 689.532631] Lustre: Skipped 2 previous similar messages [ 689.537243] Lustre: Skipped 2 previous similar messages [ 694.761565] LustreError: 19913:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 694.774392] LustreError: 19913:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 694.837540] Lustre: server umount lustre-MDT0000 complete [ 699.490357] LustreError: 19895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789683775 with bad export cookie 1244327118706906304 [ 699.496848] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 699.502866] LustreError: 19895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 699.875865] Lustre: server umount lustre-MDT0001 complete [ 709.514549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 715.804485] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 725.305646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 730.746428] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 738.228766] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 743.288384] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 745.474345] Lustre: Failing over lustre-MDT0000 [ 745.874926] Lustre: server umount lustre-MDT0000 complete [ 746.471329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 746.477122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 746.500699] Lustre: Skipped 1 previous similar message [ 761.832408] Lustre: 3635:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789683822/real 1789683822] req@ffff9bd347cc7480 x1876618868093056/t0(0) o400->MGC192.168.202.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789683838 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 761.853431] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 761.892588] LustreError: 23249:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 761.919087] LustreError: 23249:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 765.607044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 772.099088] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1144bcebf6e91871 [ 772.114414] Lustre: MGC192.168.202.112@tcp: Connection restored to 0@lo (at 0@lo) [ 772.136670] Lustre: Skipped 1 previous similar message [ 772.705108] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 777.619848] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 777.703796] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 777.714838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 777.778831] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 787.954555] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 792.037066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 792.048647] Lustre: Skipped 2 previous similar messages [ 806.369730] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 806.601103] Lustre: server umount lustre-MDT0000 complete [ 810.572294] LustreError: 23237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789683886 with bad export cookie 1244327118706907249 [ 810.597497] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 810.866339] Lustre: server umount lustre-MDT0001 complete [ 819.910355] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 18:24:54 (1789683894) [ 838.194216] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 852.815586] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 899.344421] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 910.667047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 911.040268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 911.070128] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 911.365868] Lustre: lustre-MDT0000: new disk, initializing [ 911.533371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 911.550720] Lustre: Skipped 3 previous similar messages [ 911.569598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 917.389171] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 929.779432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 929.900366] Lustre: 29651:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 930.121063] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 935.746898] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 947.171341] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 953.003644] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 959.266754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 959.604391] Lustre: 31622:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 960.998300] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 961.071715] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 961.157625] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 961.200272] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 965.011334] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 974.732276] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 980.244730] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1001.957538] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1001.970547] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1001.979152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1001.979163] Lustre: Skipped 4 previous similar messages [ 1002.017790] Lustre: Skipped 3 previous similar messages [ 1005.359854] Lustre: server umount lustre-OST0000 complete [ 1012.193164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1017.078776] Lustre: server umount lustre-MDT0000 complete [ 1017.317336] LustreError: 30530:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1017.365226] LustreError: 30530:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 1022.169401] LustreError: 31621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684098 with bad export cookie 1244327118706907662 [ 1022.180920] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1022.193656] LustreError: 31621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 1022.933706] Lustre: server umount lustre-MDT0001 complete [ 1034.385958] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 1037.379320] Key type lgssc unregistered [ 1037.841261] LNet: 33600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.868491] LNetError: 33600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.883951] LNet: Removed LNI 192.168.202.112@tcp [ 1038.709683] Key type .llcrypt unregistered [ 1038.715259] Key type ._llcrypt unregistered [ 1067.696460] Key type ._llcrypt registered [ 1067.698973] Key type .llcrypt registered [ 1067.791188] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 1069.180801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1069.423069] alg: No test for adler32 (adler32-zlib) [ 1070.627493] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 1070.963269] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 1072.679788] Key type lgssc registered [ 1074.263349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1082.852346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1082.867487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1084.331203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1088.401348] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1094.632721] LustreError: 35013:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1094.657618] LustreError: 35013:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1097.884143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1098.474176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1103.175840] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1110.168164] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1116.102885] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1123.132752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1123.235754] LustreError: 36832:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1123.571193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1128.954533] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1130.020444] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1130.180636] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1137.481311] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1153.184609] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 18:30:27 (1789684227) [ 1155.407635] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1157.524495] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1165.233183] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 18:30:39 (1789684239) [ 1175.264286] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 18:30:49 (1789684249) [ 1176.490347] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1185.836183] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1185.840376] Lustre: Skipped 2 previous similar messages [ 1187.813649] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 18:31:02 (1789684262) [ 1191.566350] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1191.571739] Lustre: Skipped 3 previous similar messages [ 1211.905821] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1211.918301] Lustre: Skipped 6 previous similar messages [ 1214.789702] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 18:31:29 (1789684289) [ 1218.017360] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1221.603115] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1231.587071] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 18:31:45 (1789684305) [ 1233.699286] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1233.706715] Lustre: Skipped 3 previous similar messages [ 1249.703104] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 18:32:03 (1789684323) [ 1272.289488] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1272.298289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1272.344933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1273.314777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1273.342512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1273.343924] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1273.405588] Lustre: Skipped 1 previous similar message [ 1278.449885] LustreError: 38248:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1278.498767] LustreError: 38248:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1278.772671] Lustre: server umount lustre-OST0000 complete [ 1288.164112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1288.174955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1288.191742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1293.795297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1293.805735] Lustre: Skipped 1 previous similar message [ 1298.399340] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1298.817620] Lustre: server umount lustre-MDT0000 complete [ 1298.928246] LustreError: 35008:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1298.939226] LustreError: 35008:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1303.095447] LustreError: 36856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684379 with bad export cookie 13944375921476498547 [ 1303.109062] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1303.110899] LustreError: 36856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1303.440315] Lustre: server umount lustre-MDT0001 complete [ 1317.643938] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 1320.801157] Key type lgssc unregistered [ 1321.082816] LNet: 42870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1321.087454] LNetError: 42870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1322.148927] LNet: Removed LNI 192.168.202.112@tcp [ 1323.084642] Key type .llcrypt unregistered [ 1323.086623] Key type ._llcrypt unregistered [ 1344.918031] Key type ._llcrypt registered [ 1344.921811] Key type .llcrypt registered [ 1345.077334] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 1361.108947] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 1362.669323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1362.708727] alg: No test for adler32 (adler32-zlib) [ 1363.959868] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 1364.307852] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 1366.015261] Key type lgssc registered [ 1367.136801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1408.613939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1408.647767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1410.029539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1410.070901] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1410.177344] Lustre: lustre-MDT0000: new disk, initializing [ 1410.261835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1410.331749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1414.812245] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1426.328849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1426.447656] Lustre: 46025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1426.517077] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1426.520084] Lustre: Skipped 1 previous similar message [ 1426.610519] Lustre: lustre-MDT0001: new disk, initializing [ 1426.719954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1426.775020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1426.789691] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1431.642299] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1441.964967] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1449.022924] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1456.955093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1457.233596] Lustre: lustre-OST0000: new disk, initializing [ 1457.237979] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1457.244592] Lustre: 47994:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1457.326916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1462.876802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1462.891362] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1463.023696] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1464.420700] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1475.692143] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1481.751405] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1481.965461] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1487.937151] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1488.130790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1493.985863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1493.992312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1494.017250] Lustre: Skipped 1 previous similar message [ 1499.110526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1499.114385] Lustre: Skipped 2 previous similar messages [ 1504.253672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1504.262357] Lustre: Skipped 1 previous similar message [ 1505.248377] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1505.538379] Lustre: server umount lustre-OST0000 complete [ 1513.953787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1513.967630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1513.984444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1519.588359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1519.598692] Lustre: Skipped 3 previous similar messages [ 1524.191141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1524.358996] Lustre: server umount lustre-MDT0000 complete [ 1524.709536] LustreError: 46034:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1524.741959] LustreError: 46034:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1528.063420] LustreError: 47993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684604 with bad export cookie 5546208180551376451 [ 1528.065032] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1528.072677] LustreError: 47993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1528.508404] Lustre: server umount lustre-MDT0001 complete [ 1540.695718] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 18:36:55 (1789684615) [ 1548.051953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1548.520890] LustreError: 50458:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1548.595102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1552.917505] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1553.889060] LustreError: 50459:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1558.008170] LustreError: 50458:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1562.842159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1563.104895] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1563.513959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1568.837206] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1577.123134] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1583.400934] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1591.516234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1591.910218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1599.485039] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1607.955600] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1612.793383] LustreError: 52986:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1613.907430] Lustre: Modifying parameter general.timeout=1 in log params [ 1614.913940] Lustre: Modifying parameter general.timeout=2 in log params [ 1617.766975] Lustre: Modifying parameter general.timeout=5 in log params [ 1617.771255] Lustre: Skipped 2 previous similar messages [ 1621.852501] Lustre: Modifying parameter general.timeout=9 in log params [ 1621.856294] Lustre: Skipped 3 previous similar messages [ 1630.667132] Lustre: Modifying parameter general.timeout=17 in log params [ 1630.670991] Lustre: Skipped 7 previous similar messages [ 1647.512549] Lustre: Modifying parameter general.timeout=35 in log params [ 1647.516072] Lustre: Skipped 17 previous similar messages [ 1680.457713] Lustre: Modifying parameter general.timeout=69 in log params [ 1680.461222] Lustre: Skipped 33 previous similar messages [ 1738.240281] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 18:40:13 (1789684813) [ 1740.728562] Lustre: 59930:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1742.527426] Lustre: 60029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1744.634980] Lustre: 60128:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1745.570629] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 in log params [ 1745.577127] Lustre: Skipped 65 previous similar messages [ 1750.238574] Lustre: 60441:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1750.244815] Lustre: 60441:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1776.193884] Lustre: 61798:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1776.219716] LustreError: 61798:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1779.858861] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 18:40:54 (1789684854) [ 1794.019194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1794.024958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1794.032932] Lustre: Skipped 1 previous similar message [ 1794.037936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1797.104782] Lustre: server umount lustre-MDT0000 complete [ 1800.636293] LustreError: 50440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684876 with bad export cookie 5546208180551377326 [ 1800.642946] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1800.648298] LustreError: 50440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1800.675083] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1800.678941] LustreError: 51747:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1800.689731] Lustre: Skipped 2 previous similar messages [ 1800.937749] Lustre: server umount lustre-MDT0001 complete [ 1806.849086] Lustre: server umount lustre-OST0000 complete [ 1828.945100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1829.185508] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1829.363853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1833.029799] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1840.311474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1840.411477] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1840.435228] Lustre: 64328:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1844.625501] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1852.138488] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1858.064528] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1863.742877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1863.876636] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1864.113460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1864.122779] Lustre: Skipped 1 previous similar message [ 1869.625403] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1875.598619] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1876.981732] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1876.987985] Lustre: Skipped 18 previous similar messages [ 1877.018234] Lustre: 66894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1884.645372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1884.660075] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1884.683986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1884.690301] Lustre: Skipped 4 previous similar messages [ 1888.500879] Lustre: server umount lustre-OST0000 complete [ 1895.395447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1895.413366] Lustre: Skipped 2 previous similar messages [ 1906.143166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1906.282718] Lustre: server umount lustre-MDT0000 complete [ 1909.610561] LustreError: 65071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684985 with bad export cookie 5546208180551380385 [ 1909.615139] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1909.640025] LustreError: 65071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1909.903823] Lustre: server umount lustre-MDT0001 complete [ 1918.265088] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 1920.451688] Key type lgssc unregistered [ 1920.726667] LNet: 68099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.732551] LNetError: 68099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1920.742248] LNet: Removed LNI 192.168.202.112@tcp [ 1921.431573] Key type .llcrypt unregistered [ 1921.434508] Key type ._llcrypt unregistered [ 1940.099176] Key type ._llcrypt registered [ 1940.104141] Key type .llcrypt registered [ 1940.188703] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 1941.082335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1941.109923] alg: No test for adler32 (adler32-zlib) [ 1942.090756] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 1942.284955] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 1943.984120] Key type lgssc registered [ 1945.012570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1951.993863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1952.008524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1953.407221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1957.266177] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1962.977883] LustreError: 69488:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1962.991115] LustreError: 69488:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1964.441904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1964.887358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1969.023559] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1975.211295] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1979.609827] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1984.749653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1985.333199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1991.077676] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1997.683672] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2004.728792] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2011.106360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2011.111748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.120679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2011.258635] Lustre: server umount lustre-MDT0000 complete [ 2014.181149] LustreError: 69489:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2014.185238] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2014.253987] LustreError: 69469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685090 with bad export cookie 15451287772373260664 [ 2014.255687] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2014.263840] LustreError: 69469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2016.524335] Lustre: server umount lustre-MDT0001 complete [ 2035.281241] Lustre: server umount lustre-OST0000 complete [ 2056.932415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2057.142341] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2057.164211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2057.370286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2060.830594] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2068.168072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2068.221110] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2068.239182] Lustre: 74357:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2068.547483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2072.236920] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2078.205809] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2083.032198] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2088.237788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2088.373768] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2088.609883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2093.831725] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2099.389651] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2100.593823] Lustre: 76931:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2102.005715] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2107.361744] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2107.364527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2107.372846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2107.382176] Lustre: Skipped 2 previous similar messages [ 2107.397907] Lustre: Skipped 1 previous similar message [ 2111.568446] Lustre: server umount lustre-OST0000 complete [ 2114.532418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2114.537782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2114.546743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2117.603593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2117.607551] Lustre: Skipped 1 previous similar message [ 2120.355477] Lustre: server umount lustre-MDT0000 complete [ 2122.724969] LustreError: 76131:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2122.735495] LustreError: 76131:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2123.156687] LustreError: 74349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685199 with bad export cookie 15451287772373261469 [ 2123.161128] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2123.170660] LustreError: 74349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2123.467857] Lustre: server umount lustre-MDT0001 complete [ 2131.224406] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 2133.240150] Key type lgssc unregistered [ 2133.444661] LNet: 78139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2133.451866] LNetError: 78139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2134.505565] LNet: Removed LNI 192.168.202.112@tcp [ 2135.140070] Key type .llcrypt unregistered [ 2135.142719] Key type ._llcrypt unregistered [ 2164.037457] Key type ._llcrypt registered [ 2164.039897] Key type .llcrypt registered [ 2164.129291] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 2165.360759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2166.016089] alg: No test for adler32 (adler32-zlib) [ 2167.301178] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 2167.571027] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 2169.239541] Key type lgssc registered [ 2170.339528] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2183.104865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2183.122742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2185.429791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2192.495995] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2194.930751] LustreError: 79551:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2194.947371] LustreError: 79551:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2200.036425] LustreError: 79550:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2204.436655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2205.034700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2209.494211] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2218.401917] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2224.656554] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2232.570451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2233.130885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2240.216294] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2247.223567] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2263.247067] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2264.015223] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2265.593691] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2265.597577] Lustre: Skipped 1 previous similar message [ 2267.886512] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2267.890181] Lustre: Skipped 2 previous similar messages [ 2272.430216] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2272.433989] Lustre: Skipped 4 previous similar messages [ 2280.672639] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2280.677552] Lustre: Skipped 8 previous similar messages [ 2297.038507] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2297.043629] Lustre: Skipped 18 previous similar messages [ 2329.817838] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2329.822330] Lustre: Skipped 37 previous similar messages [ 2356.952682] Lustre: 87677:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2359.544339] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 18:50:34 (1789685434) [ 2361.071944] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2362.725797] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 18:50:37 (1789685437) [ 2368.484904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2368.491361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2368.503584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2368.511593] Lustre: Skipped 1 previous similar message [ 2368.523955] Lustre: Skipped 1 previous similar message [ 2372.443393] Lustre: server umount lustre-OST0000 complete [ 2378.720640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2378.721665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2378.730222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2378.749084] Lustre: Skipped 1 previous similar message [ 2382.935317] Lustre: server umount lustre-MDT0000 complete [ 2383.845539] LustreError: 79545:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2383.863820] LustreError: 79545:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2386.123117] LustreError: 79530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685462 with bad export cookie 9988987033080841386 [ 2386.129289] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2386.131683] LustreError: 79530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2386.417035] Lustre: server umount lustre-MDT0001 complete [ 2397.121715] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 2399.807187] Key type lgssc unregistered [ 2400.059841] LNet: 88924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2400.065273] LNetError: 88924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2400.073883] LNet: Removed LNI 192.168.202.112@tcp [ 2400.906730] Key type .llcrypt unregistered [ 2400.915258] Key type ._llcrypt unregistered [ 2405.138325] Key type ._llcrypt registered [ 2405.140763] Key type .llcrypt registered [ 2405.286248] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2409.227210] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules [ 2410.312251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2410.340652] alg: No test for adler32 (adler32-zlib) [ 2411.342101] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 2411.586164] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 2413.256918] Key type lgssc registered [ 2414.226823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2415.401286] LustreError: 90122:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2431.524350] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 2437.625924] LustreError: 90122:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2440.250687] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2440.272406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2441.822765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2445.236902] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2451.936620] LustreError: 91147:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2451.954643] LustreError: 91147:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2453.048272] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 18:52:08 (1789685528) [ 2457.060654] LustreError: 91148:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2457.168265] LustreError: 91179:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2457.180095] LustreError: 91179:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2457.185319] LustreError: 91179:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 2457.483535] Lustre: server umount lustre-MDT0000 complete [ 2470.369194] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 2472.615322] Key type lgssc unregistered [ 2472.905534] LNet: 92670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2472.924310] LNetError: 92670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2472.940958] LNet: Removed LNI 192.168.202.112@tcp [ 2473.723689] Key type .llcrypt unregistered [ 2473.726619] Key type ._llcrypt unregistered [ 2489.283890] Key type ._llcrypt registered [ 2489.289415] Key type .llcrypt registered [ 2489.383175] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 2490.329474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2490.373489] alg: No test for adler32 (adler32-zlib) [ 2491.455186] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 2491.647894] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 2493.336043] Key type lgssc registered [ 2494.240639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2500.460120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2500.477894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2501.876596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2505.565309] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2512.353014] LustreError: 93882:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.367556] LustreError: 93882:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2512.624832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2513.163268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2516.858521] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2523.473210] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2532.255533] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2542.776169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2544.283802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2553.163757] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2561.677918] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2611.147800] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 18:54:45 (1789685685) [ 2613.605237] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2615.728765] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 18:54:50 (1789685690) [ 2620.384582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2620.395843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2620.415911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2624.484754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2624.486710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2624.508896] Lustre: Skipped 1 previous similar message [ 2624.522235] Lustre: Skipped 2 previous similar messages [ 2626.269952] Lustre: server umount lustre-MDT0000 complete [ 2629.605667] LustreError: 93882:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2629.623025] LustreError: 93882:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2630.062731] LustreError: 93862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685706 with bad export cookie 381814812940216761 [ 2630.070635] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2630.071170] LustreError: 93862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2630.478838] Lustre: server umount lustre-MDT0001 complete [ 2644.562989] Lustre: server umount lustre-OST0000 complete [ 2661.688981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2662.956252] LustreError: 98173:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2663.465301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2668.518161] LustreError: 98174:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2668.556915] LustreError: 98174:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2670.179807] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2673.640349] LustreError: 98173:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2681.136328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2681.633455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2686.310483] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2696.338539] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2706.365143] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2723.409679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2723.766309] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2723.775617] LustreError: 98164:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2723.819413] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2723.834467] LustreError: 100262:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2723.843402] LustreError: 100262:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2723.854041] LustreError: 100262:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2724.018836] Lustre: server umount lustre-OST0000 complete [ 2724.020963] LustreError: 100262:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2734.202126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2734.375543] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2734.385227] LustreError: 98161:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2734.399544] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2734.403528] LustreError: 100728:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2734.407898] LustreError: 100728:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2734.410856] LustreError: 100728:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2734.465557] Lustre: server umount lustre-OST0000 complete [ 2734.468224] LustreError: 100728:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2743.703798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2743.936174] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2743.944596] Lustre: Found index 0 for lustre-OST0000, updating log [ 2743.952336] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2744.022348] Lustre: lustre-OST0000: new disk, initializing [ 2744.029974] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2744.049399] Lustre: 101263:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2744.425309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2750.721116] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2754.487074] Lustre: Failing over lustre-OST0000 [ 2754.600103] Lustre: server umount lustre-OST0000 complete [ 2762.208639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.219901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2762.243610] Lustre: Skipped 1 previous similar message [ 2762.270090] Lustre: Skipped 1 previous similar message [ 2764.624109] Lustre: server umount lustre-MDT0000 complete [ 2767.329751] LustreError: 98173:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2767.345546] LustreError: 98173:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2768.425440] LustreError: 100746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685844 with bad export cookie 381814812940219505 [ 2768.426661] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2768.439093] LustreError: 100746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2778.787943] Lustre: server umount lustre-MDT0001 complete [ 2787.443841] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 18:57:42 (1789685862) [ 2806.210990] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 2816.616369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2816.915415] LustreError: 103611:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2816.997616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2821.493753] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2831.961254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2837.040051] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2840.121454] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2847.027835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2847.631648] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2853.349575] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2856.883588] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2856.903643] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2856.912752] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2863.914513] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2864.044879] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2864.129183] Lustre: lustre-OST0001: new disk, initializing [ 2864.132760] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2864.144731] Lustre: 105928:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2864.706624] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2864.714689] Lustre: Skipped 2 previous similar messages [ 2871.903710] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2873.909918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2873.916729] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2874.003308] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2879.760701] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2888.766449] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2894.314075] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2894.314766] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2894.338564] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2894.348231] Lustre: Skipped 2 previous similar messages [ 2899.427167] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2899.438243] Lustre: Skipped 3 previous similar messages [ 2900.071437] Lustre: server umount lustre-MDT0001 complete [ 2904.549420] LustreError: 103608:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2904.580209] LustreError: 103608:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2911.454922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2911.663613] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2911.669080] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2911.797525] Lustre: 103601:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2913.385811] LustreError: 107461:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2913.398510] LustreError: 107461:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2913.405511] LustreError: 103603:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2913.692830] LustreError: 107473:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.112@tcp: cfg command failed: rc = -17 [ 2913.733186] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2913.733186] [ 2918.395753] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2919.076083] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2931.475854] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 19:00:05 (1789686005) [ 2954.563926] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 2976.992878] Lustre: 109567:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3006.105207] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3010.998638] Lustre: 110716:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3121.073044] Lustre: server umount lustre-MDT0000 complete [ 3122.158810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3122.175485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3122.201939] Lustre: Skipped 2 previous similar messages [ 3122.226329] LustreError: 104824:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3122.256547] LustreError: 104824:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 3127.551324] LustreError: 106647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789686203 with bad export cookie 381814812940220506 [ 3127.561383] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3127.582164] LustreError: 106647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3128.433934] Lustre: server umount lustre-MDT0001 complete [ 3135.224398] Lustre: server umount lustre-OST0000 complete [ 3140.033673] Lustre: server umount lustre-OST0001 complete [ 3146.447083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3157.692027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3165.985802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3175.870212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3194.456421] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 3204.352868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3204.379704] Lustre: lustre-MDT0000: reset Object Index mappings [ 3204.755409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3204.759215] Lustre: Skipped 1 previous similar message [ 3208.756799] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3216.231493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3216.247840] Lustre: lustre-MDT0001: reset Object Index mappings [ 3220.231848] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3222.615926] Lustre: 115840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3227.989253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3229.496790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3234.947311] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3240.957056] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3249.183604] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3255.342869] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3255.475925] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3259.629619] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3273.753937] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3280.175065] Lustre: 117721:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3341.291828] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 19:06:56 (1789686416) [ 3367.913638] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3367.920663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3367.941304] Lustre: Skipped 8 previous similar messages [ 3367.949347] Lustre: Skipped 4 previous similar messages [ 3372.812908] Lustre: server umount lustre-MDT0000 complete [ 3373.031817] LustreError: 114688:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3373.059364] LustreError: 114688:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 21 previous similar messages [ 3376.571874] LustreError: 117029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789686452 with bad export cookie 381814812940267434 [ 3376.578832] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3376.580791] LustreError: 117029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3383.143781] Lustre: server umount lustre-MDT0001 complete [ 3393.742789] Lustre: server umount lustre-OST0000 complete [ 3404.437633] Lustre: server umount lustre-OST0001 complete [ 3412.796576] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 3425.263609] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 3473.507442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3473.733231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3473.771162] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3473.889956] Lustre: lustre-MDT0000: new disk, initializing [ 3474.068835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3474.073176] Lustre: Skipped 3 previous similar messages [ 3474.140109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3478.805049] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3486.137364] Lustre: server umount lustre-MDT0000 complete [ 3494.823092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3495.190824] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3495.204806] Lustre: 121993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3495.216875] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3499.261295] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3503.768990] Lustre: server umount lustre-MDT0000 complete [ 3513.306344] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 19:09:48 (1789686588) [ 3515.076186] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3516.877491] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 19:09:51 (1789686591) [ 3532.008704] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 3540.349416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3544.639717] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3551.434459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3551.543056] Lustre: 124043:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3551.612043] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3551.617759] Lustre: Skipped 1 previous similar message [ 3551.681520] Lustre: lustre-MDT0001: new disk, initializing [ 3551.837864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3551.849682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3555.486566] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3559.514335] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3565.380543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3565.683319] Lustre: lustre-OST0000: new disk, initializing [ 3565.688670] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3565.693309] Lustre: 125622:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3567.594359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3567.601627] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3567.712056] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3570.909228] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3579.735114] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3579.891732] Lustre: lustre-OST0001: new disk, initializing [ 3579.894874] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3579.899808] Lustre: 126493:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3581.981101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3581.996741] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3582.153263] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3584.968734] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3593.038725] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3596.375717] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3899.295762] Lustre: 132893:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3899.304186] Lustre: 132893:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3899.308368] Lustre: 132893:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3899.313645] Lustre: 132893:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3899.321043] Lustre: 132893:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3899.325937] Lustre: 132893:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4099.498428] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:19:33 (1789687173) [ 4120.037426] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4120.044338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4120.053912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4120.069442] Lustre: Skipped 4 previous similar messages [ 4120.100542] Lustre: Skipped 7 previous similar messages [ 4123.090503] Lustre: server umount lustre-MDT0000 complete [ 4125.240756] LustreError: 124048:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4125.266901] LustreError: 124048:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4128.723408] LustreError: 124036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789687205 with bad export cookie 381814812940337497 [ 4128.747780] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4130.279963] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4130.310046] Lustre: Skipped 1 previous similar message [ 4130.320878] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4130.335751] Lustre: Skipped 1 previous similar message [ 4136.081600] Lustre: server umount lustre-MDT0001 complete [ 4155.359147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4155.580731] Lustre: server umount lustre-OST0000 complete [ 4174.303205] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4174.507955] Lustre: server umount lustre-OST0001 complete [ 4183.631477] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 4197.725866] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 4236.771190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4237.014532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4237.050702] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4237.155487] Lustre: lustre-MDT0000: new disk, initializing [ 4237.291635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4237.297151] Lustre: Skipped 5 previous similar messages [ 4237.314208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4241.940662] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4253.709580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4253.840246] Lustre: 135946:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4253.944713] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4253.949272] Lustre: Skipped 1 previous similar message [ 4254.052982] Lustre: lustre-MDT0001: new disk, initializing [ 4254.322881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4254.335813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4258.862595] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4269.705201] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4275.840987] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4281.966399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4282.370763] Lustre: lustre-OST0000: new disk, initializing [ 4282.376377] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4282.386727] Lustre: 137916:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4284.042492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4284.057721] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4284.266530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4290.482622] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4301.051483] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4311.104091] Lustre: lustre-MDD0000: changelog on [ 4314.120712] Lustre: lustre-MDD0001: changelog on [ 6371.635233] Lustre: lustre-MDD0001: changelog off [ 6373.572330] Lustre: 137371:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6374.369454] Lustre: lustre-MDD0000: changelog off [ 6383.591569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6383.592556] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6383.601253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6383.601261] Lustre: Skipped 2 previous similar messages [ 6383.612088] Lustre: Skipped 2 previous similar messages [ 6386.789557] Lustre: server umount lustre-OST0000 complete [ 6393.828563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6393.835260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6393.846274] Lustre: Skipped 1 previous similar message [ 6393.861139] Lustre: Skipped 2 previous similar messages [ 6398.240908] Lustre: server umount lustre-MDT0000 complete [ 6398.946759] LustreError: 135958:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6398.969750] LustreError: 135958:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6402.921392] LustreError: 135939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789689479 with bad export cookie 381814812941108477 [ 6402.928576] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6402.928943] LustreError: 135939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6403.580203] Lustre: server umount lustre-MDT0001 complete [ 6418.987417] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 6423.607379] Key type lgssc unregistered [ 6423.915336] LNet: 147081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6423.925849] LNetError: 147081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6423.940594] LNet: Removed LNI 192.168.202.112@tcp [ 6424.869458] Key type .llcrypt unregistered [ 6424.871513] Key type ._llcrypt unregistered [ 6449.590899] Key type ._llcrypt registered [ 6449.594215] Key type .llcrypt registered [ 6449.800704] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 6471.013486] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 6472.575198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6472.717971] alg: No test for adler32 (adler32-zlib) [ 6473.764640] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 6474.015392] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 6475.751261] Key type lgssc registered [ 6477.658399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6521.369766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6521.396042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6522.732923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6522.803592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6522.926995] Lustre: lustre-MDT0000: new disk, initializing [ 6523.058397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6523.097049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6528.022945] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6540.707564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6540.794561] Lustre: 150283:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6540.839052] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6540.842828] Lustre: Skipped 1 previous similar message [ 6540.889886] Lustre: lustre-MDT0001: new disk, initializing [ 6540.961644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6540.997837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6541.010980] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6546.718220] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6556.896983] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6564.673083] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6573.022742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6573.472676] Lustre: lustre-OST0000: new disk, initializing [ 6573.485765] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6573.496825] Lustre: 152250:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6573.627273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6580.328897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6580.348925] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6580.533428] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6581.898655] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6595.414825] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6602.239135] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_module obdecho/obdecho [ 6604.641830] LustreError: 153265:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6604.645901] LustreError: 153265:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6606.764735] LustreError: 153361:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6621.673581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6621.698937] Lustre: Skipped 1 previous similar message [ 6621.701202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6621.717142] Lustre: Skipped 2 previous similar messages [ 6626.793842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6626.809830] Lustre: Skipped 1 previous similar message [ 6627.704184] Lustre: server umount lustre-MDT0000 complete [ 6632.347705] LustreError: 150275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789689708 with bad export cookie 9894837918906833869 [ 6632.364041] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6632.942401] Lustre: server umount lustre-MDT0001 complete [ 6648.646461] Lustre: server umount lustre-OST0000 complete [ 6675.068233] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 6690.002699] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 6737.192233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6737.546285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6737.597745] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6737.738722] Lustre: lustre-MDT0000: new disk, initializing [ 6737.807790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6737.836216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6744.904168] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6760.089864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6760.332234] Lustre: 157097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6760.434220] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6760.439844] Lustre: Skipped 1 previous similar message [ 6760.556781] Lustre: lustre-MDT0001: new disk, initializing [ 6760.788476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6760.886348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6760.894371] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6766.385070] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6778.833831] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6786.911409] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6795.246170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6795.516543] Lustre: lustre-OST0000: new disk, initializing [ 6795.521207] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6795.531182] Lustre: 159068:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6795.616839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6802.906150] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6803.027236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6803.048865] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6803.201493] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6814.748957] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6820.704757] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6821.061328] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6825.775995] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6826.102421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6829.025032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6829.042919] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6829.059184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6829.069372] Lustre: Skipped 1 previous similar message [ 6834.165560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6834.173612] Lustre: Skipped 1 previous similar message [ 6835.176844] Lustre: server umount lustre-OST0000 complete [ 6842.854611] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6842.882280] LustreError: Skipped 1 previous similar message [ 6842.897204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6842.930583] Lustre: Skipped 1 previous similar message [ 6842.959314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6847.357116] Lustre: server umount lustre-MDT0000 complete [ 6849.505229] LustreError: 158130:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6849.518853] LustreError: 158130:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6851.610533] LustreError: 157089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789689927 with bad export cookie 9894837918906834821 [ 6851.621842] LustreError: 157089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6851.630721] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6852.039475] Lustre: server umount lustre-MDT0001 complete [ 6856.644849] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 20:05:31 (1789689931) [ 6858.431818] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6860.539696] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 20:05:34 (1789689934) [ 6870.450841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6871.330986] LustreError: 161481:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6871.475254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6876.643863] LustreError: 161482:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6877.466053] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6880.744637] LustreError: 161481:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6885.860737] LustreError: 161482:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6888.313628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6894.251463] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6904.944383] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6912.271928] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6922.014859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6922.593362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6922.596577] Lustre: Skipped 1 previous similar message [ 6931.095831] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6940.807869] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6950.369663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6950.380688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6950.397314] Lustre: Skipped 1 previous similar message [ 6950.401866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6950.404659] Lustre: Skipped 2 previous similar messages [ 6953.501529] Lustre: server umount lustre-MDT0000 complete [ 6956.519912] LustreError: 161481:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6956.542269] LustreError: 161481:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6958.186340] LustreError: 161462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789690034 with bad export cookie 9894837918906835710 [ 6958.197349] LustreError: 161462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6958.214196] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6961.645469] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6961.653227] Lustre: Skipped 2 previous similar messages [ 6964.814796] Lustre: server umount lustre-MDT0001 complete [ 6973.110340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6997.880918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6998.055985] LustreError: 165329:0:(llog.c:1655:llog_backup()) MGC192.168.202.112@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6998.059985] Lustre: 165329:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.112@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7001.936783] LustreError: 165334:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7001.959986] LustreError: 165334:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7037.922169] LustreError: 165335:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7037.950370] LustreError: 165335:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 21 previous similar messages [ 7059.425436] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7059.434065] Lustre: 165329:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.112@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7099.905231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7103.018091] LustreError: 165334:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7103.035930] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7103.037358] LustreError: 165334:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 45 previous similar messages [ 7105.744178] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7117.642859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7128.551610] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7134.210770] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7135.214291] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7144.005579] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7145.892646] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7152.827152] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7154.501303] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7180.259749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7180.261496] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7180.275917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7180.275927] Lustre: Skipped 7 previous similar messages [ 7180.315438] Lustre: Skipped 1 previous similar message [ 7184.619982] Lustre: server umount lustre-MDT0000 complete [ 7189.569870] LustreError: 164269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789690265 with bad export cookie 9894837918906842164 [ 7189.572522] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7189.596954] LustreError: 164269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7190.393311] Lustre: server umount lustre-MDT0001 complete [ 7205.079380] Lustre: server umount lustre-OST0000 complete [ 7234.748734] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 7252.450278] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 7303.649213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7304.024291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7304.069928] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7304.256759] Lustre: lustre-MDT0000: new disk, initializing [ 7304.378442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7304.390770] Lustre: Skipped 1 previous similar message [ 7304.404788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7312.214680] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7328.861045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7329.034287] Lustre: 170461:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7329.749989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7335.192137] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7347.744829] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7358.808459] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7371.847931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7372.509720] Lustre: lustre-OST0000: new disk, initializing [ 7372.516829] Lustre: Skipped 1 previous similar message [ 7372.536484] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7372.539311] Lustre: Skipped 2 previous similar messages [ 7372.575688] Lustre: 172433:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7380.122966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7380.139231] Lustre: Skipped 1 previous similar message [ 7380.159550] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7380.324490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7383.111871] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7396.180933] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7401.416865] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7401.574507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7406.911987] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7407.227164] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7411.170489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7411.173688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7411.190789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7411.195134] Lustre: Skipped 2 previous similar messages [ 7411.223811] Lustre: Skipped 3 previous similar messages [ 7416.271105] Lustre: server umount lustre-OST0000 complete [ 7427.117703] Lustre: server umount lustre-MDT0000 complete [ 7431.648900] LustreError: 170474:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7431.669650] LustreError: 170474:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 25 previous similar messages [ 7431.821635] LustreError: 170455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789690508 with bad export cookie 9894837918906858999 [ 7431.823673] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7431.832600] LustreError: 170455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7432.505850] Lustre: server umount lustre-MDT0001 complete [ 7443.506257] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 20:15:17 (1789690517) [ 7452.225911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7458.808913] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7468.629747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7474.770321] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7482.497555] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7488.980443] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7497.044440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7505.751819] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7515.469728] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7528.950452] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 20:16:43 (1789690603) [ 7534.567766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7534.573494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7534.593028] Lustre: Skipped 2 previous similar messages [ 7534.600626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7534.603302] LustreError: Skipped 1 previous similar message [ 7534.621723] Lustre: Skipped 5 previous similar messages [ 7540.563508] Lustre: server umount lustre-OST0000 complete [ 7559.135430] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7559.769470] Lustre: server umount lustre-MDT0000 complete [ 7564.258087] LustreError: 174828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789690640 with bad export cookie 9894837918906859881 [ 7564.274621] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7564.601775] Lustre: server umount lustre-MDT0001 complete [ 7578.691149] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 7581.995926] Key type lgssc unregistered [ 7582.479834] LNet: 178583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7582.487737] LNetError: 178583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7582.499651] LNet: Removed LNI 192.168.202.112@tcp [ 7583.876063] Key type .llcrypt unregistered [ 7583.880762] Key type ._llcrypt unregistered [ 7607.988736] Key type ._llcrypt registered [ 7607.992657] Key type .llcrypt registered [ 7608.112433] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 7609.462301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7609.542033] alg: No test for adler32 (adler32-zlib) [ 7610.724342] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 7611.085278] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 7612.876837] Key type lgssc registered [ 7614.099996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7623.200887] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7623.232479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7638.815409] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7643.935542] LustreError: 179835:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.112@tcp: failed processing log, type 4: rc = -110 [ 7649.120517] LustreError: 179835:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7649.131906] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7649.137415] LustreError: MGC192.168.202.112@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7649.148568] LustreError: 179835:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7649.159865] LustreError: 179835:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 7649.169733] LustreError: 179835:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 7649.270716] Lustre: server umount lustre-OST0000 complete [ 7649.274839] LustreError: 179835:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7659.816606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7661.111986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7666.884364] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7671.267669] LustreError: 180334:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7671.308347] LustreError: 180334:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7676.389826] LustreError: 180335:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7676.490420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7677.091761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7682.848462] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7692.983110] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7701.573213] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7711.227748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7711.666831] LustreError: 182150:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7711.690774] Lustre: 182150:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.112@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7711.904191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7721.775140] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7734.025889] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7738.336360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7738.346633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7738.397588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7741.925479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7741.927404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7741.960423] Lustre: Skipped 2 previous similar messages [ 7744.636599] Lustre: server umount lustre-MDT0000 complete [ 7747.042892] LustreError: 180334:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7747.077736] LustreError: 180334:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7749.937703] LustreError: 180315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789690826 with bad export cookie 1641998477598270133 [ 7749.951225] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7749.955617] LustreError: 180315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7750.718505] Lustre: server umount lustre-MDT0001 complete [ 7766.163607] Lustre: server umount lustre-OST0000 complete [ 7777.546772] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 7791.487697] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 7841.095452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7841.714888] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7841.753873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7841.898959] Lustre: lustre-MDT0000: new disk, initializing [ 7842.069365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7842.120847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7850.166748] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7864.781513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7864.885652] Lustre: 185684:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7864.956946] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7864.960344] Lustre: Skipped 1 previous similar message [ 7865.079973] Lustre: lustre-MDT0001: new disk, initializing [ 7865.226114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7865.271329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7865.288256] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7871.293527] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7883.652824] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7890.303994] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7898.872001] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7899.248435] Lustre: lustre-OST0000: new disk, initializing [ 7899.252449] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7899.262955] Lustre: 187653:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7899.382871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7907.215358] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7907.935195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7907.951631] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7908.167971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7918.418875] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7923.596413] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7923.910511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7929.351454] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7929.630585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7933.922613] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7933.938617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7933.951161] Lustre: Skipped 1 previous similar message [ 7933.952207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7939.066699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7939.073170] Lustre: Skipped 2 previous similar messages [ 7944.172205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7944.181821] Lustre: Skipped 1 previous similar message [ 7946.207180] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7946.497217] Lustre: server umount lustre-OST0000 complete [ 7952.352424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7952.367659] LustreError: Skipped 1 previous similar message [ 7952.369707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7952.396209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7957.893827] Lustre: server umount lustre-MDT0000 complete [ 7959.527400] LustreError: 187661:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7959.541034] LustreError: 187661:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7962.947641] LustreError: 185676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789691039 with bad export cookie 1641998477598270861 [ 7962.951079] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7962.955714] LustreError: 185676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7963.545777] Lustre: server umount lustre-MDT0001 complete [ 7974.822445] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 20:24:09 (1789691049) [ 7997.800391] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 8001.128949] Key type lgssc unregistered [ 8001.633204] LNet: 190602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8001.641754] LNetError: 190602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8001.660920] LNet: Removed LNI 192.168.202.112@tcp [ 8002.719731] Key type .llcrypt unregistered [ 8002.722336] Key type ._llcrypt unregistered [ 8023.967177] Key type ._llcrypt registered [ 8023.968831] Key type .llcrypt registered [ 8024.089728] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 8025.553713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8025.836278] alg: No test for adler32 (adler32-zlib) [ 8026.906404] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 8027.232105] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 8028.903721] Key type lgssc registered [ 8030.171125] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8038.595871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8038.626228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8040.067705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8044.941861] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 8050.656706] LustreError: 191817:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8050.667286] LustreError: 191817:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8053.986439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8054.524661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8059.031608] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 8068.287035] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8076.557439] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8085.275402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8085.495352] LustreError: 193688:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8085.573613] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8085.580896] LustreError: 193682:0:(llog.c:1696:llog_backup()) MGC192.168.202.112@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8085.592655] Lustre: 193682:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.112@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8085.817640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8093.409302] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 8103.203407] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8105.957048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8105.964242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8105.981844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8109.542035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8109.550447] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8109.565769] Lustre: Skipped 1 previous similar message [ 8114.661666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8114.675273] Lustre: Skipped 4 previous similar messages [ 8119.788627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8120.287332] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8120.623954] Lustre: server umount lustre-MDT0000 complete [ 8124.897827] LustreError: 191816:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8124.951263] LustreError: 191816:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8124.982312] LustreError: 193709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789691201 with bad export cookie 6031725396682082680 [ 8124.999754] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8125.001028] LustreError: 193709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8125.673959] Lustre: server umount lustre-MDT0001 complete [ 8141.055194] Lustre: server umount lustre-OST0000 complete [ 8154.375948] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 8165.247424] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 8206.835592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8207.097804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8207.162976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8207.251782] Lustre: lustre-MDT0000: new disk, initializing [ 8207.352484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8207.375471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8211.981758] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 8222.454337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8222.569691] Lustre: 197210:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8222.616875] LustreError: 197221:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8222.651479] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8222.653842] Lustre: Skipped 1 previous similar message [ 8222.656894] LustreError: 197993:0:(llog.c:1696:llog_backup()) MGC192.168.202.112@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8222.662682] LustreError: 197993:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8222.666737] Lustre: 197993:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.112@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8222.681127] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8222.685188] Lustre: Skipped 1 previous similar message [ 8222.745571] Lustre: lustre-MDT0001: new disk, initializing [ 8222.822545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8222.854112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8222.860327] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8227.148795] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 8238.234817] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8245.220561] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8252.865268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8253.145298] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8253.148344] Lustre: Skipped 1 previous similar message [ 8253.151561] LustreError: 199155:0:(llog.c:1696:llog_backup()) MGC192.168.202.112@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8253.161286] LustreError: 199155:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8253.166589] Lustre: 199155:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.112@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8253.172973] Lustre: 199155:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8253.209358] Lustre: lustre-OST0000: new disk, initializing [ 8253.216296] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8253.222972] Lustre: 199178:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8253.356438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8259.194195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8259.211303] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8259.338699] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8260.084269] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 8271.545289] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8277.117693] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8277.383577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8285.743839] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8286.031929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8290.273564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8290.283566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8290.284683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8290.284689] Lustre: Skipped 2 previous similar messages [ 8290.295662] Lustre: Skipped 1 previous similar message [ 8290.307917] LustreError: Skipped 1 previous similar message [ 8294.434758] Lustre: server umount lustre-OST0000 complete [ 8299.488111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8299.491671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8299.505874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8299.508467] Lustre: Skipped 1 previous similar message [ 8304.460380] Lustre: server umount lustre-MDT0000 complete [ 8305.637185] LustreError: 197222:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8305.670070] LustreError: 197222:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8308.583948] LustreError: 198005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789691384 with bad export cookie 6031725396682083401 [ 8308.587451] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8308.594880] LustreError: 198005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8309.053927] Lustre: server umount lustre-MDT0001 complete [ 8318.461643] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 20:29:53 (1789691393) [ 8336.431109] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 8347.382064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8348.035054] LustreError: 202307:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8348.090180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8354.037158] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 8364.984517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8371.691893] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 8376.114181] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8386.873434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8387.435934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8387.451029] Lustre: Skipped 1 previous similar message [ 8396.020656] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 8408.571669] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8409.123697] Lustre: lustre-OST0001: new disk, initializing [ 8409.126554] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8409.155672] Lustre: 204620:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8415.503916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8415.525200] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8415.632842] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8417.854107] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 8429.450625] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8436.210596] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8438.404800] Lustre: DEBUG MARKER: ADD OST3 [ 8443.575160] loop: module loaded [ 8443.623588] loop0: detected capacity change from 0 to 204800000 [ 8443.773630] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8443.796319] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8443.816988] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8449.984797] loop0: detected capacity change from 0 to 204800000 [ 8450.891769] Lustre: DEBUG MARKER: STOP OST3 [ 8456.363909] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8456.624370] Lustre: lustre-OST0003: new disk, initializing [ 8456.628246] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8456.632890] Lustre: 206606:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8456.757516] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8456.763597] Lustre: Skipped 1 previous similar message [ 8458.206760] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8458.300193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8458.306522] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8458.338195] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8460.264118] LustreError: 206744:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8461.599458] Lustre: Failing over lustre-OST0003 [ 8462.061744] Lustre: server umount lustre-OST0003 complete [ 8463.352507] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8463.359628] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8463.362945] LustreError: 203796:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8463.362956] LustreError: 203796:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8463.441369] Lustre: Skipped 2 previous similar messages [ 8463.878538] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 8475.814860] Lustre: DEBUG MARKER: START OST3 again [ 8484.186844] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8484.723290] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8486.075380] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8486.557024] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8486.560600] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8491.696795] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 8513.577461] Lustre: Failing over lustre-OST0003 [ 8513.646522] Lustre: server umount lustre-OST0003 complete [ 8515.565764] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8515.598506] LustreError: 203797:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8515.599618] Lustre: Skipped 1 previous similar message [ 8515.632864] LustreError: 203797:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 8534.501893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8534.516622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8534.536859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8534.547229] Lustre: Skipped 2 previous similar messages [ 8540.050663] Lustre: server umount lustre-MDT0000 complete [ 8544.321983] LustreError: 202290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789691620 with bad export cookie 6031725396682084283 [ 8544.333151] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8544.335775] LustreError: 202290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8551.217620] Lustre: server umount lustre-MDT0001 complete [ 8556.287187] Lustre: server umount lustre-OST0000 complete [ 8561.332741] Lustre: server umount lustre-OST0001 complete [ 8569.454966] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 8579.529727] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 8622.093850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8622.454112] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8622.512602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8622.603137] Lustre: lustre-MDT0000: new disk, initializing [ 8622.671284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8622.676327] Lustre: Skipped 1 previous similar message [ 8622.690768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8627.466349] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 8638.006276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8638.091559] Lustre: 212079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8638.475657] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8643.019097] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 8653.166271] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8659.839502] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8667.373667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8667.606343] Lustre: lustre-OST0000: new disk, initializing [ 8667.608237] Lustre: Skipped 1 previous similar message [ 8667.611565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8667.615179] Lustre: Skipped 2 previous similar messages [ 8667.619402] Lustre: 214049:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8669.054035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8669.067719] Lustre: Skipped 1 previous similar message [ 8669.075576] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8669.221369] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8674.332523] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 8685.292800] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8690.475249] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8690.694697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8695.211680] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8695.469077] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8699.884112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8699.887101] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8699.895690] Lustre: Skipped 5 previous similar messages [ 8699.903431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8699.909108] Lustre: Skipped 8 previous similar messages [ 8703.863968] Lustre: server umount lustre-OST0000 complete [ 8722.399130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8722.951992] Lustre: server umount lustre-MDT0000 complete [ 8725.472713] LustreError: 212086:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8725.484293] LustreError: 212086:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 17 previous similar messages [ 8726.989411] LustreError: 212072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789691803 with bad export cookie 6031725396682086579 [ 8726.999989] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8727.000704] LustreError: 212072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8727.645265] Lustre: server umount lustre-MDT0001 complete [ 8731.950790] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 20:36:46 (1789691806) [ 8751.544169] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 8765.776610] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 8810.241857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8810.552978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8810.587979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8810.709127] Lustre: lustre-MDT0000: new disk, initializing [ 8810.816567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8816.460613] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 8829.301036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8829.452728] Lustre: 218677:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8829.804869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8835.531412] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 8847.428894] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8854.847854] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8862.385430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8862.706584] Lustre: 220649:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8863.960070] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8864.195579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8870.548408] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 8881.618904] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8887.318688] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8887.478721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8892.991649] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8893.210582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8900.069706] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8900.077009] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8900.088922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8900.088936] Lustre: Skipped 8 previous similar messages [ 8900.090575] LustreError: Skipped 1 previous similar message [ 8900.141187] Lustre: Skipped 3 previous similar messages [ 8901.956657] Lustre: server umount lustre-OST0000 complete [ 8913.639970] Lustre: server umount lustre-MDT0000 complete [ 8915.425448] LustreError: 218684:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8915.453321] LustreError: 218684:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8918.179970] LustreError: 218671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789691994 with bad export cookie 6031725396682087461 [ 8918.188389] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8918.194027] LustreError: 218671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8918.667300] Lustre: server umount lustre-MDT0001 complete [ 8928.546970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8929.326801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8929.340153] Lustre: Skipped 5 previous similar messages [ 8935.227301] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 8944.740556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8949.809715] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 8958.406881] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8964.883769] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8973.018201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8980.435836] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 8988.480555] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9000.422846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9000.438191] LustreError: Skipped 1 previous similar message [ 9004.373550] Lustre: server umount lustre-OST0000 complete [ 9015.053148] Lustre: server umount lustre-MDT0000 complete [ 9019.201837] LustreError: 222849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789692095 with bad export cookie 6031725396682088336 [ 9019.221102] LustreError: 222849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9019.232547] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9019.654697] Lustre: server umount lustre-MDT0001 complete [ 9129.140450] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 9131.865590] Key type lgssc unregistered [ 9132.130639] LNet: 226531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9132.147832] LNetError: 226531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9132.165395] LNet: Removed LNI 192.168.202.112@tcp [ 9133.073262] Key type .llcrypt unregistered [ 9133.075313] Key type ._llcrypt unregistered [ 9160.342949] Key type ._llcrypt registered [ 9160.347353] Key type .llcrypt registered [ 9160.480701] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 9177.127982] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 9178.191315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9178.360136] alg: No test for adler32 (adler32-zlib) [ 9179.556286] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 9179.834545] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 9181.559614] Key type lgssc registered [ 9182.751673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9218.050769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9218.074252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9219.413380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9219.434903] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9219.494835] Lustre: lustre-MDT0000: new disk, initializing [ 9219.562191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9219.579606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9224.084457] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 9236.729528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9237.033747] Lustre: 229884:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9237.181873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9237.196741] Lustre: Skipped 1 previous similar message [ 9237.472279] Lustre: lustre-MDT0001: new disk, initializing [ 9237.690324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9237.719254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9237.729229] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9246.883208] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 9261.271443] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9268.509396] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9277.237767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9277.517893] Lustre: lustre-OST0000: new disk, initializing [ 9277.521250] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9277.527413] Lustre: 231854:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9277.619320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9285.168342] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 9286.231157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9286.246577] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9286.431520] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9295.765397] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9300.340965] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9300.533293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9305.174848] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9305.467850] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9312.230966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9312.240630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9312.260306] Lustre: Skipped 1 previous similar message [ 9312.272377] Lustre: Skipped 1 previous similar message [ 9313.861761] Lustre: server umount lustre-OST0000 complete [ 9319.913769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9319.930500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9319.967160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9322.468185] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9322.469216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9322.506312] Lustre: Skipped 1 previous similar message [ 9324.484397] Lustre: server umount lustre-MDT0000 complete [ 9328.590821] LustreError: 233395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789692404 with bad export cookie 18020611781702617831 [ 9328.599971] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9328.605603] LustreError: 233395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9329.076849] Lustre: server umount lustre-MDT0001 complete [ 9353.524498] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 9365.459120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9365.987837] LustreError: 234789:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9366.003544] LustreError: 234789:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9366.076731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9370.751961] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 9371.119586] LustreError: 234790:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9371.150383] LustreError: 234790:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9376.224244] LustreError: 234789:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9379.485507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9379.859018] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9385.007165] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 9388.656347] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9396.735470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9397.238720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9405.371768] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 9417.175667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9417.454136] Lustre: lustre-OST0001: new disk, initializing [ 9417.457633] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9417.462533] Lustre: 237100:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9417.600497] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9426.041982] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 9426.099625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9426.114937] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9426.248830] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9438.391862] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9444.823898] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9458.731472] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 20:48:52 (1789692532) [ 9461.733238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9461.741439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9461.772786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9467.363089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9467.375912] Lustre: Skipped 5 previous similar messages [ 9467.772452] Lustre: server umount lustre-MDT0000 complete [ 9471.909483] LustreError: 234770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789692548 with bad export cookie 18020611781702618713 [ 9471.936176] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9472.494166] Lustre: server umount lustre-MDT0001 complete [ 9487.371112] Lustre: server umount lustre-OST0000 complete [ 9488.165476] Lustre: 228103:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789692548/real 1789692548] req@ffff9bd2416ea300 x1876628347143552/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789692564 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9488.213907] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9488.232048] Lustre: Skipped 3 previous similar messages [ 9491.731544] Lustre: server umount lustre-OST0001 complete [ 9499.908032] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 9512.627686] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 9559.565663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9559.842626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9559.867377] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9559.976471] Lustre: lustre-MDT0000: new disk, initializing [ 9560.093054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9560.140150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9565.417937] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 9577.475059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9577.646991] Lustre: 241048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9577.735989] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9577.742361] Lustre: Skipped 1 previous similar message [ 9577.871381] Lustre: lustre-MDT0001: new disk, initializing [ 9578.072163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9578.084894] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9583.760985] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 9594.781561] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9602.035096] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9610.015066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9610.435337] Lustre: lustre-OST0000: new disk, initializing [ 9610.443643] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9610.458765] Lustre: 243019:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9612.460578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9612.472279] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9612.642739] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9617.687828] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 9628.712577] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9634.722892] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9634.926861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9640.322950] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9640.578608] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9643.489520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9643.497969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9643.526382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9648.855293] Lustre: server umount lustre-OST0000 complete [ 9659.721955] Lustre: server umount lustre-MDT0000 complete [ 9663.968263] LustreError: 241054:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9664.000284] LustreError: 241054:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9664.093511] LustreError: 241041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789692740 with bad export cookie 18020611781702620848 [ 9664.093989] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9664.102781] LustreError: 241041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9664.483423] Lustre: server umount lustre-MDT0001 complete [ 9674.494713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9675.015586] LustreError: 245239:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9675.113605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9675.119940] Lustre: Skipped 2 previous similar messages [ 9679.967538] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 9685.476055] LustreError: 245240:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9685.507493] LustreError: 245240:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9690.161753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9696.479996] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 9705.568368] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9712.725281] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9720.507908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9727.707583] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 9735.537806] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9884.163876] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9884.193068] Lustre: Skipped 4 previous similar messages [ 9884.196872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9884.201627] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9884.215153] Lustre: Skipped 8 previous similar messages [ 9884.228347] LustreError: Skipped 1 previous similar message [ 9887.695278] Lustre: server umount lustre-OST0000 complete [ 9906.143101] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9906.314713] Lustre: server umount lustre-MDT0000 complete [ 9909.734534] LustreError: 245239:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9909.753159] LustreError: 245239:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9910.488563] LustreError: 248062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789692986 with bad export cookie 18020611781702621723 [ 9910.490836] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9910.500501] LustreError: 248062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9910.849831] Lustre: server umount lustre-MDT0001 complete [10007.020336] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [10011.074853] Key type lgssc unregistered [10011.498719] LNet: 248848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10011.525780] LNetError: 248848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10011.546321] LNet: Removed LNI 192.168.202.112@tcp [10012.666200] Key type .llcrypt unregistered [10012.668641] Key type ._llcrypt unregistered [10045.274031] Key type ._llcrypt registered [10045.275692] Key type .llcrypt registered [10045.387609] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [10063.137546] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [10064.629860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10064.756165] alg: No test for adler32 (adler32-zlib) [10065.806268] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [10066.067041] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [10067.807546] Key type lgssc registered [10068.975392] Lustre: Echo OBD driver; http://www.lustre.org/ [10125.841443] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [10139.188545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10139.284558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10140.671984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10140.699785] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10140.855252] Lustre: lustre-MDT0000: new disk, initializing [10140.986485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10141.021911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10145.778863] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [10158.390274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10158.525299] Lustre: 252940:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10158.575391] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10158.579705] Lustre: Skipped 1 previous similar message [10158.648189] Lustre: lustre-MDT0001: new disk, initializing [10158.732064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10158.792183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10158.812046] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10163.265714] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [10168.174455] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10175.702668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10176.001630] Lustre: lustre-OST0000: new disk, initializing [10176.011680] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10176.021506] Lustre: 254571:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10176.122458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10181.704363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10181.718747] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10181.928597] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10183.829254] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [10196.120412] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10196.273802] Lustre: lustre-OST0001: new disk, initializing [10196.277079] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10196.285363] Lustre: 255454:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10196.431601] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10204.828814] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [10205.835121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10205.842134] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10205.977191] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10216.934997] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10226.674234] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10240.992624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10241.000894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10241.025308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10242.020791] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10242.028480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10242.031331] Lustre: Skipped 2 previous similar messages [10242.043929] Lustre: Skipped 3 previous similar messages [10246.789247] Lustre: server umount lustre-MDT0000 complete [10251.127710] LustreError: 252932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789693327 with bad export cookie 8772616894901238737 [10251.134627] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10251.138829] LustreError: 252932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10251.730145] Lustre: server umount lustre-MDT0001 complete [10266.688771] Lustre: server umount lustre-OST0000 complete [10268.640175] Lustre: 250423:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789693328/real 1789693328] req@ffff9bd24c8aa680 x1876629276146304/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789693344 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10268.672672] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10270.787598] Lustre: server umount lustre-OST0001 complete [10286.034880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10297.726259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10307.435615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10320.561843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10344.819105] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [10355.594605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10355.627340] Lustre: lustre-MDT0000: reset Object Index mappings [10356.119554] LustreError: 260467:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10356.132194] LustreError: 260467:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10356.288319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10361.311573] LustreError: 260468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10362.338118] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [10366.433924] LustreError: 260467:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10371.591278] LustreError: 260468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10371.984685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10372.004022] Lustre: lustre-MDT0001: reset Object Index mappings [10372.391080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10377.243565] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [10380.309569] Lustre: 261608:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10387.823990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10395.419958] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [10397.477728] LustreError: 261963:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10401.600539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10404.657376] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10405.044165] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10405.049440] Lustre: Skipped 1 previous similar message [10410.518811] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10412.157327] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [10420.182115] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10424.113431] Lustre: 263478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10430.946870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10430.950600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10430.966715] Lustre: Skipped 2 previous similar messages [10430.978618] Lustre: Skipped 3 previous similar messages [10431.579257] Lustre: server umount lustre-MDT0000 complete [10435.439546] LustreError: 260448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789693511 with bad export cookie 8772616894901245520 [10435.444322] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10435.450565] LustreError: 260448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10435.835259] Lustre: server umount lustre-MDT0001 complete [10450.799163] Lustre: server umount lustre-OST0000 complete [10452.262900] Lustre: 250422:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789693512/real 1789693512] req@ffff9bd24bfe4700 x1876629276197120/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789693528 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10452.292283] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10452.300580] Lustre: Skipped 1 previous similar message [10455.298918] Lustre: server umount lustre-OST0001 complete [10477.935035] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 21:05:52 (1789693552) [10496.602996] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [10507.455413] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [10542.285826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10542.572902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10542.593353] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10542.669649] Lustre: lustre-MDT0000: new disk, initializing [10542.738873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10542.771826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10547.412953] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [10557.977343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10558.186715] Lustre: 267734:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10558.263987] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10558.267931] Lustre: Skipped 1 previous similar message [10558.376842] Lustre: lustre-MDT0001: new disk, initializing [10558.514352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10558.524719] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10562.977887] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [10573.205776] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10579.929720] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10586.465442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10586.728942] Lustre: lustre-OST0000: new disk, initializing [10586.732206] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10586.737465] Lustre: 269706:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10588.357619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10588.369527] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10588.462494] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10593.105586] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [10601.367264] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10606.019231] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10606.289509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10611.101598] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10611.281684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10614.240598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10614.242981] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10614.249167] LustreError: Skipped 1 previous similar message [10614.253770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10614.294903] Lustre: Skipped 1 previous similar message [10619.179949] Lustre: server umount lustre-OST0000 complete [10624.481810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10624.492057] Lustre: Skipped 2 previous similar messages [10628.830231] Lustre: server umount lustre-MDT0000 complete [10629.601863] LustreError: 267741:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10629.638149] LustreError: 267741:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10632.218669] LustreError: 267727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789693708 with bad export cookie 8772616894901247620 [10632.222400] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10632.231111] LustreError: 267727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10632.612786] Lustre: server umount lustre-MDT0001 complete [10651.537754] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [10661.305449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10661.598224] LustreError: 272641:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10661.639190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10661.646490] Lustre: Skipped 2 previous similar messages [10665.800794] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [10673.574361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10679.128726] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [10682.015792] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10688.504466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10694.689623] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [10703.234258] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10703.456198] Lustre: lustre-OST0001: new disk, initializing [10703.459868] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10703.471112] Lustre: 274946:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10705.440258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10705.456285] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10705.607511] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10711.116774] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [10720.861721] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10724.088780] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10728.928858] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10729.033053] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10740.704089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10740.714110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10740.727128] Lustre: Skipped 2 previous similar messages [10740.731176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10741.417947] Lustre: server umount lustre-MDT0000 complete [10742.766498] LustreError: 272642:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10742.796539] LustreError: 272642:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10744.826935] LustreError: 274945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789693821 with bad export cookie 8772616894901248502 [10744.841222] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10744.848381] LustreError: 274945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10745.144809] Lustre: server umount lustre-MDT0001 complete [10759.188742] Lustre: server umount lustre-OST0000 complete [10764.261731] Lustre: 250421:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789693824/real 1789693824] req@ffff9bd24c31ed80 x1876629276300544/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789693840 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10765.049550] Lustre: server umount lustre-OST0001 complete [10792.712497] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [10805.425311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10806.177123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10806.180411] Lustre: Skipped 3 previous similar messages [10811.371906] LustreError: 278289:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10811.403134] LustreError: 278289:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10812.301754] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [10825.283610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10831.966270] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [10835.667038] Lustre: 279429:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10843.702253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10850.917483] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [10851.379810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10861.787341] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10867.283416] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10869.225360] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [10877.920159] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10882.481547] Lustre: 281309:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10888.647410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10888.697386] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10903.390780] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 21:12:58 (1789693978) [10922.976225] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10922.981487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10923.016240] Lustre: Skipped 4 previous similar messages [10923.025762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10926.071057] Lustre: server umount lustre-MDT0000 complete [10931.019672] LustreError: 279809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789694007 with bad export cookie 8772616894901254886 [10931.032992] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10931.034261] LustreError: 279809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10937.633544] Lustre: server umount lustre-MDT0001 complete [10948.927556] Lustre: server umount lustre-OST0000 complete [10953.771479] Lustre: server umount lustre-OST0001 complete [10963.545321] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [10980.445217] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [11035.990626] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [11047.919686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11048.209038] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11048.236408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11048.325372] Lustre: lustre-MDT0000: new disk, initializing [11048.420579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11053.593205] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [11065.688673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11065.872876] Lustre: 285859:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11066.082960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11066.091738] Lustre: Skipped 4 previous similar messages [11066.130564] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11070.649119] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [11082.875278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11083.218967] Lustre: 287491:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11085.086823] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11090.454198] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [11103.346189] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11103.590628] Lustre: 288365:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11105.485571] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11105.490599] Lustre: Skipped 1 previous similar message [11105.599565] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11111.815217] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [11122.285945] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11126.948295] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11126.951904] Lustre: Skipped 1 previous similar message [11143.869167] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 21:16:58 (1789694218) [11146.739264] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11146.754412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11146.765235] Lustre: Skipped 8 previous similar messages [11146.787083] Lustre: Skipped 11 previous similar messages [11148.774448] Lustre: server umount lustre-MDT0000 complete [11151.847993] LustreError: 287498:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11151.871681] LustreError: 287498:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [11152.587854] LustreError: 285852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789694228 with bad export cookie 8772616894901262838 [11152.597744] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11152.602978] LustreError: 285852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11153.013255] Lustre: server umount lustre-MDT0001 complete [11167.310347] Lustre: server umount lustre-OST0000 complete [11182.270796] Lustre: server umount lustre-OST0001 complete [11191.420331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11197.042226] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [11206.536954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11211.967426] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [11219.386906] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11225.596231] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11232.241951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11238.615216] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [11247.462921] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11267.632062] loop1: detected capacity change from 0 to 204800000 [11267.849349] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11267.860737] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11267.880072] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11273.364500] loop1: detected capacity change from 0 to 204800000 [11279.428627] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11279.528101] LustreError: 290605:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs30752: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11279.547561] LustreError: 290605:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs30752 [11279.556626] LustreError: 290605:0:(mgs_handler.c:517:mgs_target_reg()) fs30752-MDT0001: Error checking target index: rc = -13 [11279.565251] LustreError: fs30752-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11279.575230] LustreError: 294333:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [11279.580927] LustreError: 294333:0:(tgt_mount.c:2128:server_put_super()) no obd fs30752-MDT0001 [11279.587257] LustreError: 294333:0:(tgt_mount.c:116:server_deregister_mount()) fs30752-MDT0001 not registered [11279.617704] Lustre: server umount fs30752-MDT0001 complete [11279.625906] LustreError: 294333:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11289.529525] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11289.708681] Lustre: Setting parameter fs30752-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs30752-MDT0001 [11289.776115] Lustre: srv-fs30752-MDT0001: No data found on store. Initialize space. [11289.784036] Lustre: Skipped 4 previous similar messages [11289.884984] Lustre: fs30752-MDT0001: new disk, initializing [11289.888508] Lustre: Skipped 3 previous similar messages [11290.003735] Lustre: srv-fs30752-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11290.019364] Lustre: 294800:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs30752-MDT0001-mdtlov: can't create new llog: rc = -115 [11290.035196] LustreError: 294800:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs30752-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11294.861308] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [11301.227672] Lustre: server umount fs30752-MDT0001 complete [11315.826230] loop1: detected capacity change from 0 to 204800000 [11318.759847] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11322.647178] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11323.096844] Lustre: 296369:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs30752-MDT0001-mdtlov: can't create new llog: rc = -115 [11323.106636] LustreError: 296369:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs30752-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11327.556677] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [11332.897872] Lustre: server umount fs30752-MDT0001 complete [11334.133187] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11344.369832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11344.393067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11344.404084] Lustre: Skipped 2 previous similar messages [11349.148318] Lustre: server umount lustre-MDT0000 complete [11358.027295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11358.230776] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11362.655917] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [11363.826906] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11370.920829] loop1: detected capacity change from 0 to 204800000 [11376.600028] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11376.978841] Lustre: 298816:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs30752-MDT0001-mdtlov: can't create new llog: rc = -115 [11376.981700] LustreError: 298816:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs30752-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11381.820836] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [11386.508514] Lustre: server umount fs30752-MDT0001 complete [11409.897929] LustreError: 295447:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11409.910975] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11409.929171] LustreError: 295447:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 106 previous similar messages [11413.139315] Lustre: server umount lustre-OST0000 complete [11422.808320] Lustre: server umount lustre-MDT0000 complete [11426.255699] LustreError: 290596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789694502 with bad export cookie 8772616894901265526 [11426.258676] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11426.264021] LustreError: 290596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11426.533535] Lustre: server umount lustre-MDT0001 complete [11437.202536] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [11439.716853] Key type lgssc unregistered [11439.979619] LNet: 301158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11439.987102] LNetError: 301158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11440.000991] LNet: Removed LNI 192.168.202.112@tcp [11440.769133] Key type .llcrypt unregistered [11440.771219] Key type ._llcrypt unregistered [11459.884912] Key type ._llcrypt registered [11459.894621] Key type .llcrypt registered [11460.060163] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [11461.586042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11461.597488] alg: No test for adler32 (adler32-zlib) [11462.674590] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [11462.957925] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [11464.641659] Key type lgssc registered [11465.753820] Lustre: Echo OBD driver; http://www.lustre.org/ [11473.391329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11473.419588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11474.916854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11479.976444] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [11485.164770] LustreError: 302425:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11485.191150] LustreError: 302425:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11488.248369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11488.859702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11493.500759] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [11499.818766] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11505.336902] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11511.935521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11512.274648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11514.352930] LustreError: 304256:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11518.356894] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [11519.464298] LustreError: 304257:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11525.249634] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11525.604894] LustreError: 304257:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11525.618528] LustreError: 304257:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11530.736647] LustreError: 304257:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11530.748995] LustreError: 304257:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11540.960595] LustreError: 304257:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11540.983613] LustreError: 304257:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11541.158976] loop1: detected capacity change from 0 to 204800000 [11541.273166] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11541.278812] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11541.284217] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11545.440479] loop1: detected capacity change from 0 to 204800000 [11549.291552] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11549.428762] Lustre: fs14593-OST0000: new disk, initializing [11549.431175] Lustre: srv-fs14593-OST0000: No data found on store. Initialize space. [11549.435254] Lustre: 306158:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11549.488397] Lustre: fs14593-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11553.538299] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [11558.865275] LustreError: 304256:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11558.885466] LustreError: 304256:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [11560.091817] Lustre: server umount fs14593-OST0000 complete [11568.814662] loop1: detected capacity change from 0 to 204800000 [11573.122050] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11573.252203] Lustre: fs14593-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11578.077421] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [11585.709828] loop2: detected capacity change from 0 to 204800000 [11585.893249] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11585.901203] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11585.909128] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11590.056759] loop2: detected capacity change from 0 to 204800000 [11592.169201] LustreError: 304402:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11592.182885] LustreError: 304402:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [11594.636229] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11594.730982] LustreError: fs14593-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11594.738775] LustreError: fs14593-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11594.743341] LustreError: 309196:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [11594.748215] LustreError: 309196:0:(tgt_mount.c:2128:server_put_super()) no obd fs14593-OST0001 [11594.753157] LustreError: 309196:0:(tgt_mount.c:116:server_deregister_mount()) fs14593-OST0001 not registered [11594.782700] Lustre: server umount fs14593-OST0001 complete [11594.787091] LustreError: 309196:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11597.289630] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11602.412513] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11602.466220] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11602.562439] Lustre: fs14593-OST0001: new disk, initializing [11602.573401] Lustre: srv-fs14593-OST0001: No data found on store. Initialize space. [11602.580094] Lustre: 309747:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11602.632257] Lustre: fs14593-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11607.682853] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [11614.779551] Lustre: server umount fs14593-OST0000 complete [11621.912168] Lustre: server umount fs14593-OST0001 complete [11630.516186] loop1: detected capacity change from 0 to 204800000 [11634.245788] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11634.402887] Lustre: fs14593-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11638.415357] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [11642.580447] loop2: detected capacity change from 0 to 204800000 [11645.974790] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11650.006405] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [11655.909324] Lustre: server umount fs14593-OST0000 complete [11658.726714] LustreError: 309365:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11658.734064] LustreError: 309365:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [11662.272900] Lustre: server umount fs14593-OST0001 complete [11673.057516] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11673.066556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11673.072709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11674.086518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11674.091283] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11679.210865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11679.214923] Lustre: Skipped 1 previous similar message [11684.323118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11684.334602] Lustre: Skipped 2 previous similar messages [11685.343114] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11685.487865] Lustre: server umount lustre-OST0000 complete [11688.417953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11688.427272] LustreError: Skipped 1 previous similar message [11688.430131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11688.436778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11699.681220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11699.684598] Lustre: Skipped 4 previous similar messages [11702.239170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11702.384042] Lustre: server umount lustre-MDT0000 complete [11705.107074] LustreError: 302407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789694781 with bad export cookie 14751524858413926434 [11705.115076] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11705.117994] LustreError: 302407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11705.403670] Lustre: server umount lustre-MDT0001 complete [11714.352510] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [11716.260238] Key type lgssc unregistered [11716.486874] LNet: 315665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11716.494052] LNetError: 315665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11716.510623] LNet: Removed LNI 192.168.202.112@tcp [11717.133325] Key type .llcrypt unregistered [11717.135123] Key type ._llcrypt unregistered [11739.663592] Key type ._llcrypt registered [11739.665888] Key type .llcrypt registered [11739.749559] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [11750.405180] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [11751.133138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11751.197618] alg: No test for adler32 (adler32-zlib) [11752.144829] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [11752.344961] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [11753.983393] Key type lgssc registered [11754.763612] Lustre: Echo OBD driver; http://www.lustre.org/ [11777.714801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11777.725191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11778.857915] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11778.888315] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11778.954889] Lustre: lustre-MDT0000: new disk, initializing [11779.032474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11779.046519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11781.723295] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [11789.058121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11789.130842] Lustre: 319498:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11789.159959] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11789.162259] Lustre: Skipped 1 previous similar message [11789.206595] Lustre: lustre-MDT0001: new disk, initializing [11789.275261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11789.312688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11789.321751] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11792.472227] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [11799.510580] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11803.437442] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11807.813268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11807.980721] Lustre: lustre-OST0000: new disk, initializing [11807.982498] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11807.986155] Lustre: 321465:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11808.044919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11812.097168] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [11815.475150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11815.487605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11815.611862] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11818.611970] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11820.274982] LustreError: 317671:0:(lmv_obd.c:211:lmv_notify()) activation of lustre-MDT0001_UUID failed: -22 [11820.307312] Lustre: Mounted lustre-client - version 2.17.58_39_g12a437a [11821.030143] Lustre: Unmounted lustre-client [11830.755490] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11830.776697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11833.827075] Lustre: server umount lustre-OST0000 complete [11840.489708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11840.496516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11840.504486] Lustre: Skipped 1 previous similar message [11840.511131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11840.515661] Lustre: Skipped 1 previous similar message [11842.362499] Lustre: server umount lustre-MDT0000 complete [11844.780485] LustreError: 319488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789694921 with bad export cookie 11806813229402374580 [11844.784674] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11844.787192] LustreError: 319488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11844.972069] Lustre: server umount lustre-MDT0001 complete [11851.155692] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [11852.786769] Key type lgssc unregistered [11870.367856] LNet: 323619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11870.384301] LNetError: 323619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11949.225526] LNet: Removed LNI 192.168.202.112@tcp [11949.800287] Key type .llcrypt unregistered [11949.803448] Key type ._llcrypt unregistered [11960.229620] Key type ._llcrypt registered [11960.231354] Key type .llcrypt registered [11960.316462] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [11969.211765] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [11969.803206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11969.895059] alg: No test for adler32 (adler32-zlib) [11970.839032] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [11970.979334] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [11972.591219] Key type lgssc registered [11973.240370] Lustre: Echo OBD driver; http://www.lustre.org/ [11991.964526] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 21:31:07 (1789695067) [11995.323700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11995.335775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11996.492895] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11996.514896] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11996.561484] Lustre: lustre-MDT0000: new disk, initializing [11996.594994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11996.605724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11998.615272] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [12004.722970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12004.778995] Lustre: 326808:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12004.820538] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12004.823416] Lustre: Skipped 1 previous similar message [12004.865865] Lustre: lustre-MDT0001: new disk, initializing [12004.956502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12004.981729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12004.986222] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12006.962364] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [12012.887558] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12016.721320] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12021.399357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12021.566618] Lustre: lustre-OST0000: new disk, initializing [12021.569048] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12021.571897] Lustre: 328776:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12021.621459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12025.538799] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [12027.409517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12027.415926] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12027.476773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12031.246850] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12042.720729] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12042.725502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12042.731619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12042.734166] Lustre: Skipped 1 previous similar message [12045.863428] Lustre: server umount lustre-OST0000 complete [12050.913818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12050.919061] LustreError: Skipped 1 previous similar message [12050.923399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12050.934482] Lustre: Skipped 1 previous similar message [12050.953838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12052.966677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12052.967805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12054.578961] Lustre: server umount lustre-MDT0000 complete [12056.860996] LustreError: 326801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789695133 with bad export cookie 7786946625227053679 [12056.868197] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12057.096032] Lustre: server umount lustre-MDT0001 complete [12064.159884] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [12065.873430] Key type lgssc unregistered [12066.059749] LNet: 330683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12066.069890] LNetError: 330683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12066.084273] LNet: Removed LNI 192.168.202.112@tcp [12066.609187] Key type .llcrypt unregistered [12066.610939] Key type ._llcrypt unregistered [12078.536170] Key type ._llcrypt registered [12078.537872] Key type .llcrypt registered [12078.602120] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [12079.186253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12079.217077] alg: No test for adler32 (adler32-zlib) [12080.136507] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [12080.271891] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [12081.880344] Key type lgssc registered [12082.456114] Lustre: Echo OBD driver; http://www.lustre.org/ [12086.791345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12086.815260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12088.202075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12090.768976] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [12095.960986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12096.158800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12098.284940] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [12102.104652] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12104.742745] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12108.348521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12108.519340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12112.113549] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [12116.769841] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12126.902289] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12127.957441] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12131.301498] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12131.304904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12131.316844] Lustre: Skipped 2 previous similar messages [12131.322031] Lustre: Skipped 1 previous similar message [12133.849214] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.12@tcp (stopping) [12133.851754] Lustre: Skipped 1 previous similar message [12136.423848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12136.428178] Lustre: Skipped 2 previous similar messages [12136.546652] Lustre: server umount lustre-MDT0000 complete [12138.501420] LustreError: 331913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789695214 with bad export cookie 14667444572214311297 [12138.506686] LustreError: 331913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12138.506971] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12138.954636] LustreError: 331927:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12138.962113] LustreError: 331927:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12138.978593] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.12@tcp (stopping) [12141.539730] LustreError: 331927:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12141.540587] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12143.585221] LustreError: 331927:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12143.589418] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12143.597340] Lustre: Skipped 1 previous similar message [12144.715579] Lustre: server umount lustre-MDT0001 complete [12146.740172] Lustre: server umount lustre-OST0000 complete [12151.358382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12153.892169] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [12155.657474] Lustre: 336112:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.112@tcp [12157.034471] Lustre: server umount MGS complete [12161.681944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12161.921882] LustreError: 336683:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12161.934546] LustreError: 336683:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12161.964204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12163.957686] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [12167.137816] LustreError: 336684:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12168.505173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12168.700872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12170.779600] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [12175.204882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12179.026465] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [12183.353488] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12184.604327] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12188.024827] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12191.089496] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [12193.760036] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12193.763979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12193.776339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12193.782252] Lustre: Skipped 1 previous similar message [12197.860376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12197.864122] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12199.401630] Lustre: server umount lustre-OST0000 complete [12202.979706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12202.988371] Lustre: Skipped 1 previous similar message [12207.664487] Lustre: server umount lustre-MDT0000 complete [12208.096899] LustreError: 336684:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12208.103690] LustreError: 336684:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12209.489514] LustreError: 336663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789695285 with bad export cookie 14667444572214312144 [12209.497210] LustreError: 336663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12209.498618] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12209.669974] Lustre: server umount lustre-MDT0001 complete [12215.322883] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [12216.881918] Key type lgssc unregistered [12217.058935] LNet: 340113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12217.063880] LNetError: 340113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12217.074840] LNet: Removed LNI 192.168.202.112@tcp [12217.490625] Key type .llcrypt unregistered [12217.492254] Key type ._llcrypt unregistered [12227.348550] Key type ._llcrypt registered [12227.350020] Key type .llcrypt registered [12227.400662] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [12235.415654] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [12235.843905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12235.909073] alg: No test for adler32 (adler32-zlib) [12236.820499] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [12236.963497] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [12238.575376] Key type lgssc registered [12239.033952] Lustre: Echo OBD driver; http://www.lustre.org/ [12255.058109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12255.065988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12256.177783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12256.195122] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12256.232963] Lustre: lustre-MDT0000: new disk, initializing [12256.263941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12256.273761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12258.596189] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [12265.011993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12265.079183] Lustre: 343244:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12265.121930] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12265.124513] Lustre: Skipped 1 previous similar message [12265.166647] Lustre: lustre-MDT0001: new disk, initializing [12265.230736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12265.260868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12265.270799] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12267.625704] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [12272.717335] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12275.426671] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12278.564252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12278.694520] Lustre: lustre-OST0000: new disk, initializing [12278.696828] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12278.700217] Lustre: 345210:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12278.738868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12281.658214] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [12286.460354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12286.464394] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12286.518254] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12286.873769] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12289.200848] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12289.328133] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12291.664539] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12291.757576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12296.672351] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12296.677437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12296.680617] LustreError: Skipped 1 previous similar message [12296.682937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12296.687558] Lustre: Skipped 1 previous similar message [12296.695249] Lustre: Skipped 1 previous similar message [12298.858937] Lustre: server umount lustre-OST0000 complete [12301.295070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12301.300763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12301.308456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12306.807126] Lustre: server umount lustre-MDT0000 complete [12308.722237] LustreError: 346749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789695385 with bad export cookie 770463634596904186 [12308.725264] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12308.726706] LustreError: 346749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12308.938897] Lustre: server umount lustre-MDT0001 complete [12313.763151] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 21:36:29 (1789695389) [12322.613776] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [12327.781611] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [12345.922170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12346.031448] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12346.053846] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12346.104928] Lustre: lustre-MDT0000: new disk, initializing [12346.144350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12346.152512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12348.240095] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [12353.887331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12353.939848] Lustre: 349961:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12353.969510] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12353.972408] Lustre: Skipped 1 previous similar message [12354.015532] Lustre: lustre-MDT0001: new disk, initializing [12354.058457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12354.076962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12354.081880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12355.869635] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [12361.641945] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12365.190519] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12369.130986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12369.273469] Lustre: lustre-OST0000: new disk, initializing [12369.276550] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12369.281115] Lustre: 351927:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12369.317658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12372.472130] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [12378.218144] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12379.665281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12379.674729] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12379.742374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12405.218925] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12405.219772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12405.224078] LustreError: Skipped 1 previous similar message [12405.224924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12405.230886] Lustre: Skipped 2 previous similar messages [12405.232733] Lustre: Skipped 1 previous similar message [12407.693349] Lustre: server umount lustre-OST0000 complete [12410.341524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12410.344102] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12410.349286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12410.354951] Lustre: Skipped 1 previous similar message [12410.361913] Lustre: Skipped 1 previous similar message [12415.457534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12415.466788] Lustre: Skipped 2 previous similar messages [12424.161761] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12424.288827] Lustre: server umount lustre-MDT0000 complete [12425.700386] LustreError: 349967:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12425.723356] LustreError: 349967:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12426.272473] LustreError: 350754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789695502 with bad export cookie 770463634596905075 [12426.278741] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12426.281641] LustreError: 350754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12426.434268] Lustre: server umount lustre-MDT0001 complete [12431.955126] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [12433.315964] Key type lgssc unregistered [12433.486892] LNet: 354075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12433.491397] LNetError: 354075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12433.505520] LNet: Removed LNI 192.168.202.112@tcp [12433.942570] Key type .llcrypt unregistered [12433.945157] Key type ._llcrypt unregistered [12478.773182] Key type ._llcrypt registered [12478.774514] Key type .llcrypt registered [12478.822686] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [12478.984838] Key type .llcrypt unregistered [12478.988275] Key type ._llcrypt unregistered [12514.419360] Key type ._llcrypt registered [12514.420735] Key type .llcrypt registered [12514.468039] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [12514.620268] Key type .llcrypt unregistered [12514.622057] Key type ._llcrypt unregistered [12526.769838] Key type ._llcrypt registered [12526.771256] Key type .llcrypt registered [12526.818639] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [12526.936440] Key type .llcrypt unregistered [12526.938158] Key type ._llcrypt unregistered [12539.880938] Key type ._llcrypt registered [12539.882472] Key type .llcrypt registered [12539.931308] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [12540.073705] Key type .llcrypt unregistered [12540.079092] Key type ._llcrypt unregistered [12573.070386] Key type ._llcrypt registered [12573.071782] Key type .llcrypt registered [12573.114434] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [12573.227275] Key type .llcrypt unregistered [12573.228829] Key type ._llcrypt unregistered [12585.687859] Key type ._llcrypt registered [12585.689286] Key type .llcrypt registered [12585.738460] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [12585.861606] Key type .llcrypt unregistered [12585.863377] Key type ._llcrypt unregistered [12598.356255] Key type ._llcrypt registered [12598.357757] Key type .llcrypt registered [12598.400323] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [12598.515267] Key type .llcrypt unregistered [12598.516619] Key type ._llcrypt unregistered [12611.167536] Key type ._llcrypt registered [12611.168743] Key type .llcrypt registered [12611.220102] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [12611.372568] Key type .llcrypt unregistered [12611.373917] Key type ._llcrypt unregistered [12624.000218] Key type ._llcrypt registered [12624.001632] Key type .llcrypt registered [12624.047561] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [12624.178537] Key type .llcrypt unregistered [12624.180214] Key type ._llcrypt unregistered [12636.838205] Key type ._llcrypt registered [12636.839665] Key type .llcrypt registered [12636.895250] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [12637.037811] Key type .llcrypt unregistered [12637.039585] Key type ._llcrypt unregistered [12662.481584] Key type ._llcrypt registered [12662.482811] Key type .llcrypt registered [12662.529048] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [12662.654506] Key type .llcrypt unregistered [12662.655779] Key type ._llcrypt unregistered [12675.085953] Key type ._llcrypt registered [12675.087298] Key type .llcrypt registered [12675.137873] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [12675.290295] Key type .llcrypt unregistered [12675.291576] Key type ._llcrypt unregistered [12689.261445] Key type ._llcrypt registered [12689.262831] Key type .llcrypt registered [12689.336970] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [12689.534277] Key type .llcrypt unregistered [12689.536288] Key type ._llcrypt unregistered [12745.265773] Key type ._llcrypt registered [12745.274844] Key type .llcrypt registered [12745.370499] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [12758.648859] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [12759.831642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12759.964446] alg: No test for adler32 (adler32-zlib) [12761.056916] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [12761.280688] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [12762.991419] Key type lgssc registered [12764.309648] Lustre: Echo OBD driver; http://www.lustre.org/ [12790.104312] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12790.121529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12791.282394] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12791.305621] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12791.364728] Lustre: tst250fs-MDT0000: new disk, initializing [12791.419868] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12791.434792] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12794.719357] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [12801.148943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12801.217258] Lustre: 373939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12801.265200] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12801.268269] Lustre: Skipped 1 previous similar message [12801.308798] Lustre: tst250fs-MDT0001: new disk, initializing [12801.393042] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12801.429201] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12801.448874] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12804.182705] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [12807.521887] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12811.839531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12812.038625] Lustre: tst250fs-OST0000: new disk, initializing [12812.041684] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12812.046434] Lustre: 375570:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12812.114288] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12814.964870] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12814.974337] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12815.096926] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12815.872257] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [12823.364527] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12823.492375] Lustre: tst250fs-OST0001: new disk, initializing [12823.495494] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12823.498222] Lustre: 376440:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12823.554815] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12828.190616] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [12832.284537] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12832.337065] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12832.349805] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12832.473644] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12847.590205] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12847.604230] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12847.620427] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12853.228776] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12853.242101] Lustre: Skipped 7 previous similar messages [12853.543603] Lustre: server umount tst250fs-MDT0000 complete [12856.025762] LustreError: 373933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789695932 with bad export cookie 3547849381508134180 [12856.033106] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12856.035041] LustreError: 373933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12856.340698] Lustre: server umount tst250fs-MDT0001 complete [12869.166386] Lustre: server umount tst250fs-OST0000 complete [12881.046565] Lustre: server umount tst250fs-OST0001 complete [12895.373605] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [12901.331715] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [12923.689764] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 21:46:38 (1789695998) [12924.810656] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12936.735783] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [12942.779372] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [12967.360184] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12622 sec ========= 21:47:22 (1789696042) [12968.669712] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:47:23 (1789696043) === [12971.261324] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:47:26 (1789696046) === [13003.658433] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [13005.848786] Key type lgssc unregistered [13006.055638] LNet: 385565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13006.065451] LNetError: 385565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13006.081709] LNet: Removed LNI 192.168.202.112@tcp [13006.787154] Key type .llcrypt unregistered [13006.791505] Key type ._llcrypt unregistered