[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 534791682 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003000] x2apic enabled [ 0.003013] Switched APIC routing to physical x2apic. [ 0.004000] kvm-guest: setup PV IPIs [ 0.005000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.005000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.005026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.006017] pid_max: default: 32768 minimum: 301 [ 0.008125] LSM: Security Framework initializing [ 0.010068] Yama: becoming mindful. [ 0.011000] SELinux: Initializing. [ 0.011000] *** VALIDATE selinux *** [ 0.021857] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027325] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028350] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031147] *** VALIDATE tmpfs *** [ 0.033044] *** VALIDATE proc *** [ 0.034302] *** VALIDATE cgroup *** [ 0.035014] *** VALIDATE cgroup2 *** [ 0.036323] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037253] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039040] Spectre V2 : User space: Vulnerable [ 0.040014] Speculative Store Bypass: Vulnerable [ 0.044228] debug: unmapping init [mem 0xffffffffa3859000-0xffffffffa3860fff] [ 0.047696] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049482] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050034] ... version: 2 [ 0.051021] ... bit width: 48 [ 0.052021] ... generic registers: 4 [ 0.052992] ... value mask: 0000ffffffffffff [ 0.053019] ... max period: 00007fffffffffff [ 0.054021] ... fixed-purpose events: 3 [ 0.055016] ... event mask: 000000070000000f [ 0.056580] rcu: Hierarchical SRCU implementation. [ 0.059270] smp: Bringing up secondary CPUs ... [ 0.061694] x86: Booting SMP configuration: [ 0.062030] .... node #0, CPUs: #1 #2 #3 [ 0.073477] smp: Brought up 1 node, 4 CPUs [ 0.075020] smpboot: Max logical packages: 1 [ 0.076018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.158723] node 0 deferred pages initialised in 78ms [ 0.165084] devtmpfs: initialized [ 0.167857] x86/mm: Memory block size: 128MB [ 0.174000] gcov: version magic: 0x41383552 [ 0.177457] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.185589] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.192482] pinctrl core: initialized pinctrl subsystem [ 0.196277] [ 0.197011] ************************************************************* [ 0.202015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.207024] ** ** [ 0.211016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.212000] ** ** [ 0.218020] ** This means that this kernel is built to expose internal ** [ 0.221017] ** IOMMU data structures, which may compromise security on ** [ 0.232195] ** your system. ** [ 0.237021] ** ** [ 0.242017] ** If you see this message and you are not debugging the ** [ 0.246017] ** kernel, report this immediately to your vendor! ** [ 0.251016] ** ** [ 0.253015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.258020] ************************************************************* [ 0.265209] NET: Registered protocol family 16 [ 0.268476] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.272084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.276086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.281126] cpuidle: using governor menu [ 0.284124] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.288555] PCI: Using configuration type 1 for base access [ 0.293426] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.322114] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.324038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.340065] cryptd: max_cpu_qlen set to 1000 [ 0.344282] ACPI: Added _OSI(Module Device) [ 0.345000] ACPI: Added _OSI(Processor Device) [ 0.345000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.362024] ACPI: Added _OSI(Processor Aggregator Device) [ 0.374102] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.387667] ACPI: Interpreter enabled [ 0.389074] ACPI: PM: (supports S0 S3 S4 S5) [ 0.391018] ACPI: Using IOAPIC for interrupt routing [ 0.394115] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.399418] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.423968] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.428047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.431023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.436310] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.443896] acpiphp: Slot [2] registered [ 0.445330] acpiphp: Slot [5] registered [ 0.447207] acpiphp: Slot [6] registered [ 0.449269] acpiphp: Slot [7] registered [ 0.452371] acpiphp: Slot [8] registered [ 0.454222] acpiphp: Slot [9] registered [ 0.456137] acpiphp: Slot [10] registered [ 0.457158] acpiphp: Slot [3] registered [ 0.459129] acpiphp: Slot [4] registered [ 0.460129] acpiphp: Slot [11] registered [ 0.463177] acpiphp: Slot [12] registered [ 0.464000] acpiphp: Slot [13] registered [ 0.466134] acpiphp: Slot [14] registered [ 0.471200] acpiphp: Slot [15] registered [ 0.474415] acpiphp: Slot [16] registered [ 0.477143] acpiphp: Slot [17] registered [ 0.480425] acpiphp: Slot [18] registered [ 0.484212] acpiphp: Slot [19] registered [ 0.486217] acpiphp: Slot [20] registered [ 0.490816] acpiphp: Slot [21] registered [ 0.493420] acpiphp: Slot [22] registered [ 0.496345] acpiphp: Slot [23] registered [ 0.499250] acpiphp: Slot [24] registered [ 0.502177] acpiphp: Slot [25] registered [ 0.504127] acpiphp: Slot [26] registered [ 0.505000] acpiphp: Slot [27] registered [ 0.505328] acpiphp: Slot [28] registered [ 0.508137] acpiphp: Slot [29] registered [ 0.510126] acpiphp: Slot [30] registered [ 0.512140] acpiphp: Slot [31] registered [ 0.513085] PCI host bridge to bus 0000:00 [ 0.515025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.519024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.522026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.525030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.529026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.533037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.537000] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.545000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.548000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.576021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.582593] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.587021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.594022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.595018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.603590] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.605666] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.608045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.613493] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.624029] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.649020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.671025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.689375] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.712020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.734026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.776017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.792000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.828026] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.844023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.883019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.896000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.918022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.928018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.962020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.980000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.992020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.004023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.031017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 1.045000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.054017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.064020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.084019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.095133] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.101021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.107026] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.126018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.137000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.139364] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.141379] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.143444] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.146268] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.154046] iommu: Default domain type: Passthrough [ 1.160588] SCSI subsystem initialized [ 1.166757] ACPI: bus type USB registered [ 1.172758] usbcore: registered new interface driver usbfs [ 1.181187] usbcore: registered new interface driver hub [ 1.189091] usbcore: registered new device driver usb [ 1.191200] pps_core: LinuxPPS API ver. 1 registered [ 1.194019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.197129] PTP clock support registered [ 1.202095] EDAC MC: Ver: 3.0.0 [ 1.204203] PCI: Using ACPI for IRQ routing [ 1.206614] NetLabel: Initializing [ 1.207000] NetLabel: domain hash size = 128 [ 1.213015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.216527] NetLabel: unlabeled traffic allowed by default [ 1.220146] vgaarb: loaded [ 1.224303] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.225016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.243406] clocksource: Switched to clocksource kvm-clock [ 1.632919] VFS: Disk quotas dquot_6.6.0 [ 1.644637] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.650831] *** VALIDATE ramfs *** [ 1.656456] *** VALIDATE hugetlbfs *** [ 1.661098] pnp: PnP ACPI init [ 1.672689] pnp: PnP ACPI: found 6 devices [ 1.723912] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.733748] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.740354] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.744914] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.748813] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.752372] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.756294] NET: Registered protocol family 2 [ 1.763371] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.771963] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.776827] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.784335] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.790624] TCP: Hash tables configured (established 65536 bind 65536) [ 1.795936] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.801349] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.805844] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.814094] NET: Registered protocol family 1 [ 1.822278] RPC: Registered named UNIX socket transport module. [ 1.830338] RPC: Registered udp transport module. [ 1.834745] RPC: Registered tcp transport module. [ 1.838311] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.849736] NET: Registered protocol family 44 [ 1.856598] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.862270] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.869605] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.889833] pci 0000:00:01.0: quirk_isa_dma_hangs+0x0/0x20 took 19752 usecs [ 1.894293] PCI: CLS 0 bytes, default 64 [ 1.901760] Unpacking initramfs... [ 5.975927] debug: unmapping init [mem 0xffff8e1bfcc54000-0xffff8e1bfffbffff] [ 5.991457] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 6.005738] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 6.018585] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 7.636325] Initialise system trusted keyrings [ 7.638867] Key type blacklist registered [ 7.642830] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 7.666213] zbud: loaded [ 7.670914] *** VALIDATE nfs *** [ 7.673119] *** VALIDATE nfs4 *** [ 7.674570] pstore: using deflate compression [ 7.698658] Platform Keyring initialized [ 7.906253] NET: Registered protocol family 38 [ 7.911112] Key type asymmetric registered [ 7.913568] Asymmetric key parser 'x509' registered [ 7.923984] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 7.929542] io scheduler mq-deadline registered [ 7.931289] io scheduler kyber registered [ 7.933830] io scheduler bfq registered [ 7.936907] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 7.939369] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 7.945454] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.953787] ACPI: Power Button [PWRF] [ 7.961250] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.972489] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.991620] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 8.002494] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 8.051324] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 8.090574] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 8.136929] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 8.148967] Non-volatile memory driver v1.3 [ 8.152715] Linux agpgart interface v0.103 [ 8.208682] virtio_blk virtio1: [vda] 146680 512-byte logical blocks (75.1 MB/71.6 MiB) [ 8.212941] vda: detected capacity change from 0 to 75100160 [ 8.243192] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 8.247299] vdb: detected capacity change from 0 to 1073741824 [ 8.270774] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.274235] vdc: detected capacity change from 0 to 2621440000 [ 8.290667] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.295327] vdd: detected capacity change from 0 to 2621440000 [ 8.323088] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.330267] vde: detected capacity change from 0 to 4294967296 [ 8.361274] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.364235] vdf: detected capacity change from 0 to 4294967296 [ 8.382295] libphy: Fixed MDIO Bus: probed [ 8.425897] usbcore: registered new interface driver usbserial_generic [ 8.428766] usbserial: USB Serial support registered for generic [ 8.432664] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.438070] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.447087] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.451739] mousedev: PS/2 mouse device common for all mice [ 8.458793] rtc_cmos 00:05: RTC can wake from S4 [ 8.463388] rtc_cmos 00:05: registered as rtc0 [ 8.466398] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.471393] intel_pstate: CPU model not supported [ 8.475267] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.481432] hid: raw HID events driver (C) Jiri Kosina [ 8.481633] usbcore: registered new interface driver usbhid [ 8.481637] usbhid: USB HID core driver [ 8.481765] drop_monitor: Initializing network drop monitor service [ 8.481892] Initializing XFRM netlink socket [ 8.484954] NET: Registered protocol family 10 [ 8.493407] Segment Routing with IPv6 [ 8.493445] NET: Registered protocol family 17 [ 8.494649] mpls_gso: MPLS GSO support [ 8.511569] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 8.518666] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.519041] RAS: Correctable Errors collector initialized. [ 8.525667] AVX version of gcm_enc/dec engaged. [ 8.528663] AES CTR mode by8 optimization enabled [ 8.632854] sched_clock: Marking stable (8632835401, 0)->(10389825446, -1756990045) [ 8.638669] registered taskstats version 1 [ 8.641929] Loading compiled-in X.509 certificates [ 8.644845] zswap: loaded using pool lzo/zbud [ 8.679204] Key type big_key registered [ 8.698794] Key type encrypted registered [ 8.700350] ima: No TPM chip found, activating TPM-bypass! [ 8.703525] ima: Allocated hash algorithm: sha1 [ 8.705565] ima: No architecture policies found [ 8.709829] evm: Initialising EVM extended attributes: [ 8.713802] evm: security.selinux [ 8.715357] evm: security.ima [ 8.718319] evm: security.capability [ 8.720720] evm: HMAC attrs: 0x1 [ 8.724914] rtc_cmos 00:05: setting system clock to 2026-08-27 10:16:55 UTC (1787825815) [ 8.733432] debug: unmapping init [mem 0xffffffffa4803000-0xffffffffa49fffff] [ 8.737654] debug: unmapping init [mem 0xffffffffa3582000-0xffffffffa3858fff] [ 8.754497] Write protecting the kernel read-only data: 28672k [ 8.764153] debug: unmapping init [mem 0xffffffffa1c03000-0xffffffffa1dfffff] [ 8.770245] debug: unmapping init [mem 0xffffffffa2514000-0xffffffffa25fffff] [ 8.823799] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.841706] systemd[1]: Detected virtualization kvm. [ 8.844814] systemd[1]: Detected architecture x86-64. [ 8.850736] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.889300] systemd[1]: No hostname configured. [ 8.891961] systemd[1]: Set hostname to . [ 8.893874] random: systemd: uninitialized urandom read (16 bytes read) [ 8.897738] systemd[1]: Initializing machine ID from random generator. [ 9.041797] random: ln: uninitialized urandom read (6 bytes read) [ 9.216239] random: systemd: uninitialized urandom read (16 bytes read) [ 9.219817] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 9.224895] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 9.239489] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.507578] device-mapper: uevent: version 1.0.3 [ 10.511673] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices.[ 11.971882] random: fast init done Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 12.157024] virtio_net virtio0 ens2: renamed from eth0 [ 12.590523] scsi host0: ata_piix [ 12.710380] scsi host1: ata_piix [ 12.711525] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.716656] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 17.236192] random: crng init done [ 17.237541] random: 7 urandom warning(s) missed due to ratelimiting [ 18.598111] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 20.674193] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.656245] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.179739] SELinux: Disabled at runtime. [ 24.255907] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.274964] systemd[1]: Detected virtualization kvm. [ 24.277658] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.407300] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.411345] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.416645] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.420296] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.425735] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.438605] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.451643] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ 25.960525] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 26.892455] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.049838] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.102310] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.754583] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.836643] EDAC sbridge: Ver: 1.1.2 [ 31.859540] Key type dns_resolver registered [ 32.249198] NFS: Registering the id_resolver key type [ 32.251199] Key type id_resolver registered [ 32.253452] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg118-server login: [ 44.557025] hrtimer: interrupt took 12796491 ns [ 95.258409] libcfs: loading out-of-tree module taints kernel. [ 95.349490] Key type ._llcrypt registered [ 95.352777] Key type .llcrypt registered [ 95.482566] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 114.522486] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 116.209910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 116.222706] alg: No test for adler32 (adler32-zlib) [ 117.755103] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 118.535865] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 120.395714] Key type lgssc registered [ 122.087228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 140.568091] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 188.364170] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 201.823511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 201.860987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 203.172782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 203.193509] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 203.261943] Lustre: lustre-MDT0000: new disk, initializing [ 203.338253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 203.355762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 208.041678] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.682568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 221.821120] Lustre: 6527:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 221.843600] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 221.848812] Lustre: Skipped 1 previous similar message [ 221.941226] Lustre: lustre-MDT0001: new disk, initializing [ 222.027308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 222.071987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 222.085444] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 226.471310] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.981618] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 240.942323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 241.167454] Lustre: lustre-OST0000: new disk, initializing [ 241.170463] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 241.175329] Lustre: 8467:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 241.220450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 246.322155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 246.328891] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 246.385603] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 247.088628] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 260.905080] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 261.031540] Lustre: lustre-OST0001: new disk, initializing [ 261.036139] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 261.043830] Lustre: 9540:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 261.124332] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 267.468321] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 269.907214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 269.918182] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 270.007464] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 280.130423] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 287.353924] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 295.077822] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing check_logdir /tmp/testlogs/ [ 300.232526] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing yml_node [ 305.615621] Lustre: DEBUG MARKER: Client: 2.17.57.68 [ 308.413784] Lustre: DEBUG MARKER: MDS: 2.17.57.68 [ 312.702824] Lustre: DEBUG MARKER: OSS: 2.17.57.68 [ 315.347557] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 06:21:58 EDT 2026 [ 340.259705] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 342.460338] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 360.417046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 360.432775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.446342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 362.471025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 362.473386] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 367.586811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 367.593873] Lustre: Skipped 6 previous similar messages [ 372.705485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 372.709215] Lustre: Skipped 1 previous similar message [ 374.240155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 374.579302] Lustre: server umount lustre-MDT0000 complete [ 378.420071] LustreError: 6519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826185 with bad export cookie 4170198019196858655 [ 378.428075] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 378.429516] LustreError: 6519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 382.945404] LustreError: 6564:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 382.948324] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 382.960075] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 382.971614] LustreError: 6564:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 383.003066] Lustre: Skipped 3 previous similar messages [ 383.971180] LustreError: 6534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 384.008307] LustreError: 6534:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 384.902902] Lustre: server umount lustre-MDT0001 complete [ 388.675543] Lustre: server umount lustre-OST0000 complete [ 391.994625] Lustre: server umount lustre-OST0001 complete [ 398.666817] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 407.658840] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 442.920495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 443.171706] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 443.204700] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 443.274947] Lustre: lustre-MDT0000: new disk, initializing [ 443.428985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 443.448788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 448.362585] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 460.463737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 460.636871] Lustre: 15094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 460.715645] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 460.719689] Lustre: Skipped 1 previous similar message [ 460.897364] Lustre: lustre-MDT0001: new disk, initializing [ 461.109162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 461.171289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 461.185988] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 466.165244] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 476.593341] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 482.251782] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 489.131654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 489.386357] Lustre: lustre-OST0000: new disk, initializing [ 489.392206] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 489.398524] Lustre: 17062:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 489.476346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 490.546341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 490.555362] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 490.699747] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 496.042744] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 507.031681] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 512.638957] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 512.931622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 518.278089] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 518.512147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 521.186857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 521.191559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 521.206501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 521.212282] Lustre: Skipped 3 previous similar messages [ 527.081307] Lustre: server umount lustre-OST0000 complete [ 531.941316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.958618] Lustre: Skipped 2 previous similar messages [ 537.064144] LustreError: 15100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 537.092082] LustreError: 15100:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 537.120706] Lustre: server umount lustre-MDT0000 complete [ 540.930528] LustreError: 15087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826347 with bad export cookie 4170198019196860188 [ 540.939793] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 540.944790] LustreError: 15087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 541.516532] Lustre: server umount lustre-MDT0001 complete [ 547.434290] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 06:25:51 (1787826351) [ 568.882515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.311973] LustreError: 19935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 569.417686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 574.433796] LustreError: 19936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 575.615608] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 584.677232] LustreError: 19935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 584.697326] LustreError: 19935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 585.803292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 590.569788] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 598.227498] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 605.187987] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 607.690434] Lustre: Failing over lustre-MDT0000 [ 607.984226] Lustre: server umount lustre-MDT0000 complete [ 610.787829] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 610.826885] LustreError: 19930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 610.851046] LustreError: 19930:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 627.040204] Lustre: 3660:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826417/real 1787826417] req@ffff8e1c7d6c6580 x1874671357300992/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787826433 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 627.077649] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 630.073650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 636.322527] LustreError: 3659:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e1b45a2f840 x1874671357303936/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 636.732657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 636.737317] Lustre: Skipped 1 previous similar message [ 636.768934] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 641.644892] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 642.025403] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 642.036381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 642.055905] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 653.067515] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 660.455485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 660.469287] Lustre: Skipped 1 previous similar message [ 660.480795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 660.483756] Lustre: Skipped 5 previous similar messages [ 664.764804] Lustre: server umount lustre-MDT0000 complete [ 665.579290] LustreError: 19936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 665.611507] LustreError: 19936:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 668.222286] LustreError: 19916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826474 with bad export cookie 4170198019196861602 [ 668.225895] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 668.234430] LustreError: 19916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 668.588410] Lustre: server umount lustre-MDT0001 complete [ 680.635985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 687.760616] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 697.893666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 704.588918] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 713.439094] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 720.092838] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 722.381146] Lustre: Failing over lustre-MDT0000 [ 722.407863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 722.424882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 722.425256] Lustre: Skipped 2 previous similar messages [ 722.447916] Lustre: Skipped 1 previous similar message [ 722.616631] Lustre: server umount lustre-MDT0000 complete [ 733.666483] LustreError: 24005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 733.680456] LustreError: 24005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 739.808221] Lustre: 3661:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826530/real 1787826530] req@ffff8e1c7fb10000 x1874671357332096/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787826546 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 739.842719] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 743.263817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 750.113425] LustreError: 3659:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e1c7d5aa940 x1874671357335040/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 750.435860] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 755.441666] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 755.701891] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 755.704230] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 755.713657] Lustre: Skipped 1 previous similar message [ 755.737593] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 768.389478] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 776.161714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 776.166711] LustreError: Skipped 1 previous similar message [ 781.729687] Lustre: server umount lustre-MDT0000 complete [ 787.472768] LustreError: 23255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826594 with bad export cookie 4170198019196862540 [ 787.490168] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 787.904789] Lustre: server umount lustre-MDT0001 complete [ 798.710622] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 06:30:02 (1787826602) [ 818.389767] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 830.161107] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 882.971717] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 896.407726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 896.799452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 896.849135] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 896.984623] Lustre: lustre-MDT0000: new disk, initializing [ 897.138948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 897.148929] Lustre: Skipped 3 previous similar messages [ 897.179296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 902.990381] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 914.581045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 914.746430] Lustre: 29659:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 915.196314] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 920.884986] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 931.432202] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 938.139668] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 946.330261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 946.744088] Lustre: 31630:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 948.268384] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 948.319617] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 948.384467] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 948.417324] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 953.630237] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 964.453680] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 972.581449] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 999.917214] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 999.926864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 999.941336] Lustre: Skipped 2 previous similar messages [ 999.970057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 999.972762] Lustre: Skipped 3 previous similar messages [ 1004.513584] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1005.998920] Lustre: server umount lustre-OST0000 complete [ 1017.286912] Lustre: server umount lustre-MDT0000 complete [ 1020.385968] LustreError: 30524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1020.435287] LustreError: 30524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 1021.884136] LustreError: 29652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826828 with bad export cookie 4170198019196862946 [ 1021.906308] LustreError: 29652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1021.910619] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1022.699087] Lustre: server umount lustre-MDT0001 complete [ 1035.161414] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1038.753992] Key type lgssc unregistered [ 1039.095884] LNet: 33612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1039.104939] LNetError: 33612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1039.126817] LNet: Removed LNI 192.168.201.118@tcp [ 1040.315488] Key type .llcrypt unregistered [ 1040.317820] Key type ._llcrypt unregistered [ 1066.865155] Key type ._llcrypt registered [ 1066.873967] Key type .llcrypt registered [ 1066.986967] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1068.721479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1068.965327] alg: No test for adler32 (adler32-zlib) [ 1070.231962] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 1070.694929] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1072.376258] Key type lgssc registered [ 1073.454081] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1082.245060] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1082.268584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1083.748159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1090.350676] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1094.130260] LustreError: 35028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1094.143202] LustreError: 35028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1099.256250] LustreError: 35029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1102.234482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1102.961852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1109.313788] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1119.197718] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1127.589348] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1136.849534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1137.558092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1147.449977] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1147.825583] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1154.025308] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1156.024428] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1170.733367] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 06:36:14 (1787826974) [ 1172.687620] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1174.957202] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1184.542033] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 06:36:28 (1787826988) [ 1197.168626] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 06:36:40 (1787827000) [ 1198.485677] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1209.699903] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1209.704479] Lustre: Skipped 2 previous similar messages [ 1212.077840] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 06:36:55 (1787827015) [ 1214.487877] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1214.493878] Lustre: Skipped 2 previous similar messages [ 1224.774067] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1224.778620] Lustre: Skipped 4 previous similar messages [ 1241.659570] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 06:37:25 (1787827045) [ 1245.304365] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1249.990957] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1260.273623] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 06:37:44 (1787827064) [ 1262.689400] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1262.694320] Lustre: Skipped 6 previous similar messages [ 1280.393074] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 06:38:03 (1787827083) [ 1301.477338] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1301.486822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1301.507329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1302.503707] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1302.512922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1302.518683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1302.536365] Lustre: Skipped 1 previous similar message [ 1305.676504] Lustre: server umount lustre-OST0000 complete [ 1312.737221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1312.746429] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1312.772270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1312.791445] Lustre: Skipped 1 previous similar message [ 1316.617856] Lustre: server umount lustre-MDT0000 complete [ 1317.861894] LustreError: 35758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1317.898832] LustreError: 35758:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1321.819314] LustreError: 35009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827128 with bad export cookie 6467000960174296572 [ 1321.825729] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1321.834928] LustreError: 35009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1322.448468] Lustre: server umount lustre-MDT0001 complete [ 1335.349160] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1338.834993] Key type lgssc unregistered [ 1339.127836] LNet: 42895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1339.134915] LNetError: 42895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1340.201065] LNet: Removed LNI 192.168.201.118@tcp [ 1341.406640] Key type .llcrypt unregistered [ 1341.410336] Key type ._llcrypt unregistered [ 1363.280920] Key type ._llcrypt registered [ 1363.283962] Key type .llcrypt registered [ 1363.558652] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 1385.290591] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1387.023749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1387.395522] alg: No test for adler32 (adler32-zlib) [ 1388.615739] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 1388.937524] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1390.642108] Key type lgssc registered [ 1391.718834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1436.860490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1436.888465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1438.178451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1438.201727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1438.299380] Lustre: lustre-MDT0000: new disk, initializing [ 1438.385777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1438.404792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1444.473342] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1456.751365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1456.944748] Lustre: 46052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1457.096574] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1457.101408] Lustre: Skipped 1 previous similar message [ 1457.236200] Lustre: lustre-MDT0001: new disk, initializing [ 1457.368216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1457.402628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1457.411900] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1462.778977] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1474.678154] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1481.282815] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1489.244138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1489.670171] Lustre: lustre-OST0000: new disk, initializing [ 1489.674772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1489.683330] Lustre: 48021:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1489.797487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1497.180343] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1497.684363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1497.713693] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1497.879796] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1509.256616] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1514.872161] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1515.127361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1520.266432] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1520.496334] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1523.685742] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1523.687425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1523.704422] LustreError: Skipped 1 previous similar message [ 1523.706601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1523.734805] Lustre: Skipped 1 previous similar message [ 1528.890829] Lustre: server umount lustre-OST0000 complete [ 1533.922155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1533.928888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1533.932655] Lustre: Skipped 2 previous similar messages [ 1539.042244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1539.049143] Lustre: Skipped 1 previous similar message [ 1539.519934] Lustre: server umount lustre-MDT0000 complete [ 1544.163229] LustreError: 46065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1544.183507] LustreError: 46065:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1544.229117] LustreError: 46045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827351 with bad export cookie 17659967352356016412 [ 1544.240201] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1544.245529] LustreError: 46045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1544.544607] Lustre: server umount lustre-MDT0001 complete [ 1557.612825] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 06:42:41 (1787827361) [ 1568.077241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1568.668120] LustreError: 50483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1568.769203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1573.857675] LustreError: 50484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1574.822836] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1578.982461] LustreError: 50483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1584.105398] LustreError: 50484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1586.261680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1586.690158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1593.230858] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1603.360529] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1610.905362] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1620.510362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1620.995029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1628.380747] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1637.009109] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1646.518443] LustreError: 53015:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1647.479563] Lustre: Modifying parameter general.timeout=1 in log params [ 1648.513991] Lustre: Modifying parameter general.timeout=2 in log params [ 1650.757380] Lustre: Modifying parameter general.timeout=4 in log params [ 1650.761857] Lustre: Skipped 1 previous similar message [ 1655.444904] Lustre: Modifying parameter general.timeout=8 in log params [ 1655.464535] Lustre: Skipped 3 previous similar messages [ 1664.457444] Lustre: Modifying parameter general.timeout=16 in log params [ 1664.466842] Lustre: Skipped 7 previous similar messages [ 1681.209066] Lustre: Modifying parameter general.timeout=32 in log params [ 1681.220822] Lustre: Skipped 15 previous similar messages [ 1713.228230] Lustre: Modifying parameter general.timeout=61 in log params [ 1713.235892] Lustre: Skipped 28 previous similar messages [ 1777.299574] Lustre: Modifying parameter general.timeout=110 in log params [ 1777.304945] Lustre: Skipped 48 previous similar messages [ 1811.690519] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 06:46:55 (1787827615) [ 1814.547588] Lustre: 59982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1816.386849] Lustre: 60080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1820.251516] Lustre: 60294:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1820.261423] Lustre: 60294:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1855.507665] Lustre: 61863:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1855.522926] Lustre: 61863:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1855.565157] LustreError: 61863:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1860.716701] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 06:47:44 (1787827664) [ 1878.499101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1878.503045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1878.514279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1878.541407] Lustre: Skipped 3 previous similar messages [ 1881.506467] Lustre: server umount lustre-MDT0000 complete [ 1883.639357] LustreError: 50484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1883.669813] LustreError: 50484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1886.167700] LustreError: 50463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827692 with bad export cookie 17659967352356017301 [ 1886.180619] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1886.191221] LustreError: 50463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1886.642277] Lustre: server umount lustre-MDT0001 complete [ 1901.406295] Lustre: server umount lustre-OST0000 complete [ 1931.761802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1932.177924] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1932.230262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1932.252040] Lustre: Skipped 42 previous similar messages [ 1932.581438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1937.408284] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1946.344453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1946.549940] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1946.618840] Lustre: 64396:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1952.587769] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1962.581880] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1970.079120] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1978.283500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1978.511497] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1979.103742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1979.115812] Lustre: Skipped 1 previous similar message [ 1986.089490] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1994.351064] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1996.457648] Lustre: 66962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 2008.037895] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2008.041379] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2008.072411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2008.095775] Lustre: Skipped 2 previous similar messages [ 2009.765022] Lustre: server umount lustre-OST0000 complete [ 2018.276130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2018.294690] Lustre: Skipped 1 previous similar message [ 2018.299045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2018.315723] Lustre: Skipped 2 previous similar messages [ 2020.405865] Lustre: server umount lustre-MDT0000 complete [ 2023.405666] LustreError: 65529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2023.448067] LustreError: 65529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2025.276529] LustreError: 64388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827832 with bad export cookie 17659967352356020633 [ 2025.292453] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2025.298019] LustreError: 64388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2025.795646] Lustre: server umount lustre-MDT0001 complete [ 2038.198974] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2041.567239] Key type lgssc unregistered [ 2041.983752] LNet: 68176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2041.989803] LNetError: 68176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2043.066591] LNet: Removed LNI 192.168.201.118@tcp [ 2043.952312] Key type .llcrypt unregistered [ 2043.958394] Key type ._llcrypt unregistered [ 2071.875967] Key type ._llcrypt registered [ 2071.879783] Key type .llcrypt registered [ 2072.126954] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2073.793173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2073.957712] alg: No test for adler32 (adler32-zlib) [ 2075.182876] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 2075.621049] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2077.480598] Key type lgssc registered [ 2079.409146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2089.798777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2089.834930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2091.574343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2095.975260] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2101.729834] LustreError: 69593:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2101.762537] LustreError: 69593:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2106.849262] LustreError: 69592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2109.407287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2110.522418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2116.092818] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2125.555771] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2132.234551] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2141.019065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2141.836316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2149.906264] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2159.591589] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2172.982926] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2182.117077] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.130616] Lustre: Skipped 2 previous similar messages [ 2182.133877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2182.155757] Lustre: Skipped 2 previous similar messages [ 2187.234704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2187.237981] Lustre: Skipped 2 previous similar messages [ 2188.254510] Lustre: server umount lustre-MDT0000 complete [ 2192.087496] LustreError: 69573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827998 with bad export cookie 17163194031686258605 [ 2192.100690] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2192.110708] LustreError: 69573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2192.380764] Lustre: server umount lustre-MDT0001 complete [ 2205.560191] Lustre: server umount lustre-OST0000 complete [ 2232.703225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2233.084247] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2233.152698] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2233.449763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2239.101283] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2248.346569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2248.493071] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2248.525609] Lustre: 74469:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2249.220840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2254.467551] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2261.932222] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2267.684887] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2274.034444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2274.153204] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2274.358322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2280.868695] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2289.927539] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2292.066176] Lustre: 77044:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2294.905403] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2301.923326] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2301.933948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2301.955605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2302.450273] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2305.629957] Lustre: server umount lustre-OST0000 complete [ 2310.626061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2310.643978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2310.661170] Lustre: Skipped 1 previous similar message [ 2310.670058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2310.672244] Lustre: Skipped 2 previous similar messages [ 2312.683149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.206904] Lustre: server umount lustre-MDT0000 complete [ 2317.794741] LustreError: 74479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2317.819914] LustreError: 74479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2320.160972] LustreError: 76336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828126 with bad export cookie 17163194031686259438 [ 2320.169500] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2320.172429] LustreError: 76336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2320.395742] Lustre: server umount lustre-MDT0001 complete [ 2331.675702] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2334.283339] Key type lgssc unregistered [ 2334.520239] LNet: 78255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2334.525937] LNetError: 78255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2335.594734] LNet: Removed LNI 192.168.201.118@tcp [ 2336.474103] Key type .llcrypt unregistered [ 2336.476657] Key type ._llcrypt unregistered [ 2360.419343] Key type ._llcrypt registered [ 2360.424571] Key type .llcrypt registered [ 2360.525780] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2361.950822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2362.004335] alg: No test for adler32 (adler32-zlib) [ 2363.099166] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 2363.335685] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2364.984363] Key type lgssc registered [ 2366.416919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2375.324618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2375.348753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2376.731323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2381.479297] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2386.923553] LustreError: 79671:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.953969] LustreError: 79671:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2391.530625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2392.583370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2398.402901] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2408.500853] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2414.501016] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2422.878127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2423.689635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2431.624865] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2440.900394] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2452.896265] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2454.049229] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2455.092328] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2457.472227] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2457.477129] Lustre: Skipped 1 previous similar message [ 2461.784550] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2461.792030] Lustre: Skipped 3 previous similar messages [ 2470.594754] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2470.598992] Lustre: Skipped 7 previous similar messages [ 2487.266362] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2487.278763] Lustre: Skipped 16 previous similar messages [ 2519.350765] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2519.354894] Lustre: Skipped 33 previous similar messages [ 2563.674727] Lustre: 87781:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2567.028244] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 06:59:30 (1787828370) [ 2569.522306] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2571.815643] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 06:59:35 (1787828375) [ 2576.370470] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2576.376667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2576.402646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2576.435674] Lustre: Skipped 1 previous similar message [ 2581.478192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2581.512036] Lustre: Skipped 1 previous similar message [ 2586.596933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2586.601023] Lustre: Skipped 1 previous similar message [ 2590.176716] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2590.565968] Lustre: server umount lustre-OST0000 complete [ 2596.836042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2596.837343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2596.857917] Lustre: Skipped 1 previous similar message [ 2596.861333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2601.961727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2601.971013] Lustre: Skipped 2 previous similar messages [ 2610.144143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2610.527562] Lustre: server umount lustre-MDT0000 complete [ 2612.195543] LustreError: 82171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2612.207609] LustreError: 82171:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2614.702538] LustreError: 79650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828421 with bad export cookie 13356917587917695599 [ 2614.717080] LustreError: 79650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2614.723227] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2615.150326] Lustre: server umount lustre-MDT0001 complete [ 2628.929187] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2632.473323] Key type lgssc unregistered [ 2632.705531] LNet: 89037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2632.713911] LNetError: 89037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2632.734686] LNet: Removed LNI 192.168.201.118@tcp [ 2633.645112] Key type .llcrypt unregistered [ 2633.646474] Key type ._llcrypt unregistered [ 2640.624420] Key type ._llcrypt registered [ 2640.626871] Key type .llcrypt registered [ 2640.748705] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2646.358610] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules [ 2648.249305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2648.397846] alg: No test for adler32 (adler32-zlib) [ 2649.876721] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 2650.271425] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2651.978873] Key type lgssc registered [ 2654.341666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2657.066781] LustreError: 90273:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2674.438477] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2678.642370] LustreError: 90273:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2685.497614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2685.514804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2687.027110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2692.587435] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2697.188470] LustreError: 91253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2697.217692] LustreError: 91253:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2702.310070] LustreError: 91252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2703.983316] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 07:01:47 (1787828507) [ 2707.433901] LustreError: 91253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2709.173953] LustreError: 91284:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2709.188987] LustreError: 91284:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2709.200152] LustreError: 91284:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 2709.636318] Lustre: server umount lustre-MDT0000 complete [ 2727.058143] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2730.262445] Key type lgssc unregistered [ 2730.531829] LNet: 92783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2730.539824] LNetError: 92783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2731.559522] LNet: Removed LNI 192.168.201.118@tcp [ 2732.421288] Key type .llcrypt unregistered [ 2732.422990] Key type ._llcrypt unregistered [ 2753.903505] Key type ._llcrypt registered [ 2753.907708] Key type .llcrypt registered [ 2754.082865] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2755.357105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2755.441816] alg: No test for adler32 (adler32-zlib) [ 2756.662367] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 2757.103761] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2758.792172] Key type lgssc registered [ 2759.970635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2769.401874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2769.435929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2771.108167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2776.667690] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2780.647848] LustreError: 94001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2780.665119] LustreError: 94001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2785.769899] LustreError: 94002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2787.163996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2787.920873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2793.131794] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2803.439429] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2811.367731] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2821.239373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2822.078691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2831.514588] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2841.647266] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2896.249456] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 07:04:59 (1787828699) [ 2898.149953] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2900.285386] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 07:05:04 (1787828704) [ 2905.063461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2905.066882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2905.076561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2907.105158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2907.107446] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2907.114784] Lustre: Skipped 1 previous similar message [ 2907.127885] Lustre: Skipped 1 previous similar message [ 2911.482029] Lustre: server umount lustre-MDT0000 complete [ 2912.230401] LustreError: 94001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2912.258582] LustreError: 94001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2915.805083] LustreError: 93982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828722 with bad export cookie 17153360006293279668 [ 2915.812759] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2915.814722] LustreError: 93982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2916.126625] Lustre: server umount lustre-MDT0001 complete [ 2931.372927] Lustre: server umount lustre-OST0000 complete [ 2945.786672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2946.475452] LustreError: 98299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2946.708043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2952.168406] LustreError: 98300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2953.148828] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2961.380421] LustreError: 98300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2961.403192] LustreError: 98300:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2963.240236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2964.149523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2970.076923] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2978.516373] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2984.823674] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3001.815934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3002.036675] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3002.044154] LustreError: 98287:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3002.069723] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3002.075886] LustreError: 100384:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 3002.085467] LustreError: 100384:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 3002.104769] LustreError: 100384:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3002.212705] Lustre: server umount lustre-OST0000 complete [ 3002.218484] LustreError: 100384:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3013.285072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3013.632427] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3013.638800] LustreError: 98288:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3013.661378] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3013.673386] LustreError: 100852:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 3013.679046] LustreError: 100852:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 3013.684335] LustreError: 100852:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3013.798163] Lustre: server umount lustre-OST0000 complete [ 3013.807205] LustreError: 100852:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3025.363105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3025.631816] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3025.634456] Lustre: Found index 0 for lustre-OST0000, updating log [ 3025.645966] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 3025.739930] Lustre: lustre-OST0000: new disk, initializing [ 3025.746556] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3025.753256] Lustre: 101386:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3026.106360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3034.293508] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3038.381057] Lustre: Failing over lustre-OST0000 [ 3038.610404] Lustre: server umount lustre-OST0000 complete [ 3045.859221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3045.873825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3045.890415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3049.424847] Lustre: server umount lustre-MDT0000 complete [ 3052.002547] LustreError: 98299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3052.029460] LustreError: 98299:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3053.991062] LustreError: 98280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828860 with bad export cookie 17153360006293282419 [ 3054.000926] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3063.477850] Lustre: server umount lustre-MDT0001 complete [ 3077.204258] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 07:08:00 (1787828880) [ 3099.378417] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3113.610453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3114.065873] LustreError: 103753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3114.240330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3120.859097] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3132.179619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3138.367579] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3142.316984] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3149.276628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3149.908700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3149.912959] Lustre: Skipped 1 previous similar message [ 3149.922449] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3157.508523] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3158.138982] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3158.181538] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3158.189815] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3168.935722] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3169.066138] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3169.173518] Lustre: lustre-OST0001: new disk, initializing [ 3169.179869] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3169.189537] Lustre: 106072:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3177.460410] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3179.196859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3179.204404] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3179.360247] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3186.542394] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3193.960905] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3199.458145] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3199.461576] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3199.487281] Lustre: Skipped 1 previous similar message [ 3199.495827] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3199.498359] Lustre: Skipped 2 previous similar messages [ 3205.099515] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3205.110409] Lustre: Skipped 3 previous similar messages [ 3213.280232] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3213.687782] Lustre: server umount lustre-MDT0001 complete [ 3215.341579] LustreError: 103748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3215.354203] LustreError: 103748:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3225.221700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3225.372918] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3225.384753] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3225.497482] Lustre: 103742:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3226.416420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3226.424610] Lustre: Skipped 1 previous similar message [ 3227.502597] LustreError: 107624:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3227.509282] LustreError: 107624:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3227.523196] LustreError: 103745:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3227.676532] LustreError: 107647:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.118@tcp: cfg command failed: rc = -17 [ 3227.681915] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3227.681915] [ 3231.738431] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3232.005429] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3243.649307] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 07:10:47 (1787829047) [ 3264.744443] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3288.895612] Lustre: 109689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3318.079904] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3322.217614] Lustre: 110827:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3410.401978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3410.417908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3410.424383] Lustre: Skipped 2 previous similar messages [ 3410.438383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3410.446568] Lustre: Skipped 5 previous similar messages [ 3415.467915] Lustre: server umount lustre-MDT0000 complete [ 3416.040145] LustreError: 103748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3416.045491] LustreError: 103748:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3420.091034] LustreError: 103733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829226 with bad export cookie 17153360006293283420 [ 3420.096543] LustreError: 103733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3420.100699] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3420.409590] Lustre: server umount lustre-MDT0001 complete [ 3436.320169] Lustre: 93313:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787829227/real 1787829227] req@ffff8e1c45104000 x1874674123654272/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787829243 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3436.348718] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3436.366078] Lustre: Skipped 3 previous similar messages [ 3437.573579] Lustre: server umount lustre-OST0000 complete [ 3440.672418] Lustre: 93311:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787829231/real 1787829231] req@ffff8e1c7da15680 x1874674123654528/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787829247 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3443.086815] Lustre: server umount lustre-OST0001 complete [ 3449.818495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3460.954712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3470.418245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3481.800467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3503.863989] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3516.592922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3516.624995] Lustre: lustre-MDT0000: reset Object Index mappings [ 3517.203567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3522.571114] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3531.891270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3531.919301] Lustre: lustre-MDT0001: reset Object Index mappings [ 3538.145181] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3541.956178] Lustre: 115927:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3548.602966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3556.085797] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3557.654075] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:231 to 0x280000400:257) [ 3561.797823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3565.299783] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3571.215103] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 3571.222262] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3571.598711] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3580.476403] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3584.661263] Lustre: 117803:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3640.583765] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 07:17:24 (1787829444) [ 3665.380247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3665.388407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3665.402505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3665.407578] Lustre: Skipped 3 previous similar messages [ 3671.479080] Lustre: server umount lustre-MDT0000 complete [ 3673.571796] LustreError: 117823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3673.585596] LustreError: 117823:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 3675.951796] LustreError: 117806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829482 with bad export cookie 17153360006293330271 [ 3675.974294] LustreError: 117806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3675.978055] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3682.620350] Lustre: server umount lustre-MDT0001 complete [ 3692.635230] Lustre: server umount lustre-OST0000 complete [ 3703.000481] Lustre: server umount lustre-OST0001 complete [ 3709.674686] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 3720.356732] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3763.681972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3763.992691] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3764.037647] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3764.134754] Lustre: lustre-MDT0000: new disk, initializing [ 3764.225339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3768.695880] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3777.558038] Lustre: server umount lustre-MDT0000 complete [ 3788.992893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3789.339464] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3789.365488] Lustre: 122083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3789.379283] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3789.559541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3789.570475] Lustre: Skipped 4 previous similar messages [ 3795.944985] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3801.501817] Lustre: server umount lustre-MDT0000 complete [ 3810.909367] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 07:20:14 (1787829614) [ 3812.704406] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3815.124174] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 07:20:18 (1787829618) [ 3839.801783] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3850.782567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3857.161741] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3866.624426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3866.781745] Lustre: 124134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3866.880462] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3866.885123] Lustre: Skipped 1 previous similar message [ 3866.973122] Lustre: lustre-MDT0001: new disk, initializing [ 3867.166363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3867.185183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3871.647161] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3876.870678] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3884.525192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3884.793675] Lustre: lustre-OST0000: new disk, initializing [ 3884.797236] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3884.804852] Lustre: 125713:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3886.610269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3886.620984] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3886.789824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3891.946885] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3902.834513] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3903.011840] Lustre: lustre-OST0001: new disk, initializing [ 3903.015715] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3903.021400] Lustre: 126583:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3904.805304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3904.852963] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3905.098926] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3911.290336] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3922.070578] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3926.401693] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4335.739375] Lustre: 132999:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4335.742957] Lustre: 132999:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4335.745585] Lustre: 132999:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4335.748375] Lustre: 132999:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4335.751368] Lustre: 132999:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4335.754045] Lustre: 132999:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4616.161838] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:33:39 (1787830419) [ 4635.110469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4635.117847] Lustre: Skipped 5 previous similar messages [ 4635.126291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4635.137350] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4635.137498] Lustre: Skipped 11 previous similar messages [ 4638.232608] Lustre: server umount lustre-MDT0000 complete [ 4640.237132] LustreError: 125708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4640.255314] LustreError: 125708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4641.654885] LustreError: 124127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830448 with bad export cookie 17153360006293399179 [ 4641.673773] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4641.676902] LustreError: 124127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4645.359272] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4645.379739] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4645.393601] Lustre: Skipped 3 previous similar messages [ 4645.407221] Lustre: Skipped 1 previous similar message [ 4648.071334] Lustre: server umount lustre-MDT0001 complete [ 4667.360525] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4667.512354] Lustre: server umount lustre-OST0000 complete [ 4687.328282] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4687.783926] Lustre: server umount lustre-OST0001 complete [ 4697.454685] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 4711.542893] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4755.721762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4756.021745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4756.081847] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4756.198271] Lustre: lustre-MDT0000: new disk, initializing [ 4756.312507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4756.316850] Lustre: Skipped 4 previous similar messages [ 4756.330764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4761.836197] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4773.961180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4774.240627] Lustre: 136058:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4774.315413] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4774.319819] Lustre: Skipped 1 previous similar message [ 4774.444388] Lustre: lustre-MDT0001: new disk, initializing [ 4774.678652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4774.686860] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4780.433535] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4790.964573] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4797.026843] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4804.103703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4804.480420] Lustre: lustre-OST0000: new disk, initializing [ 4804.486382] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4804.490702] Lustre: 138025:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4806.081597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4806.086116] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4806.287949] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4811.951889] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4823.241618] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4835.138970] Lustre: lustre-MDD0000: changelog on [ 4838.212593] Lustre: lustre-MDD0001: changelog on [ 6865.173076] Lustre: lustre-MDD0001: changelog off [ 6867.206370] Lustre: 139663:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6868.282317] Lustre: lustre-MDD0000: changelog off [ 6878.196850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6878.197951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6878.219082] Lustre: Skipped 1 previous similar message [ 6878.223886] Lustre: Skipped 2 previous similar messages [ 6881.900463] Lustre: server umount lustre-OST0000 complete [ 6888.428061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6888.437356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6888.450558] Lustre: Skipped 1 previous similar message [ 6888.469574] Lustre: Skipped 2 previous similar messages [ 6893.542415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6893.550396] Lustre: Skipped 1 previous similar message [ 6894.270517] Lustre: server umount lustre-MDT0000 complete [ 6898.212263] LustreError: 136048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832704 with bad export cookie 17153360006294177271 [ 6898.220428] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6898.235104] LustreError: 136048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6898.656886] Lustre: server umount lustre-MDT0001 complete [ 6912.195874] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 6916.542140] Key type lgssc unregistered [ 6917.072482] LNet: 147193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6917.078694] LNetError: 147193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6918.118137] LNet: Removed LNI 192.168.201.118@tcp [ 6919.050147] Key type .llcrypt unregistered [ 6919.051848] Key type ._llcrypt unregistered [ 6945.884272] Key type ._llcrypt registered [ 6945.887042] Key type .llcrypt registered [ 6946.109668] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 6963.930170] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6964.795068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6964.812045] alg: No test for adler32 (adler32-zlib) [ 6965.965779] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 6966.247886] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 6967.936933] Key type lgssc registered [ 6969.343147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7008.225928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7008.258318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7009.577720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7009.627311] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7009.755454] Lustre: lustre-MDT0000: new disk, initializing [ 7009.882403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7009.915977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7014.357319] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7025.362976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7025.524704] Lustre: 150399:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7025.602725] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7025.610680] Lustre: Skipped 1 previous similar message [ 7025.709447] Lustre: lustre-MDT0001: new disk, initializing [ 7025.833612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7025.903640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7025.910826] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7031.085522] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7041.463587] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7048.113202] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7058.739861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7059.069796] Lustre: lustre-OST0000: new disk, initializing [ 7059.077298] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7059.104521] Lustre: 152370:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7059.234306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7065.715098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7065.718039] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7065.913509] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7067.022411] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7076.492960] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7083.476685] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_module obdecho/obdecho [ 7085.696605] LustreError: 153371:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7085.704170] LustreError: 153371:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7087.430533] LustreError: 153467:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7101.925457] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7101.934387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7101.942768] Lustre: Skipped 2 previous similar messages [ 7101.952089] Lustre: Skipped 2 previous similar messages [ 7104.397904] Lustre: server umount lustre-MDT0000 complete [ 7107.684319] LustreError: 150389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832914 with bad export cookie 14678343051348308759 [ 7107.696534] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7107.698607] LustreError: 150389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7108.007342] Lustre: server umount lustre-MDT0001 complete [ 7121.985905] Lustre: server umount lustre-OST0000 complete [ 7143.210767] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 7155.394432] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7201.890597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7202.313570] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7202.346626] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7202.427600] Lustre: lustre-MDT0000: new disk, initializing [ 7202.527493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7202.555307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7208.153600] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7219.449139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7219.653453] Lustre: 157184:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7219.770923] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7219.777873] Lustre: Skipped 1 previous similar message [ 7219.901451] Lustre: lustre-MDT0001: new disk, initializing [ 7220.074028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7220.128954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7220.152969] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7225.544879] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7235.702459] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7241.572976] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7249.898764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7250.160851] Lustre: lustre-OST0000: new disk, initializing [ 7250.164919] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7250.173452] Lustre: 159153:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7250.329148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7255.709173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7255.727371] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7255.892288] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7259.104948] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7270.773327] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7276.663584] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7276.909832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7283.340480] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7283.637903] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7286.756178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7286.760727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7286.767087] Lustre: Skipped 1 previous similar message [ 7286.776399] Lustre: Skipped 1 previous similar message [ 7291.895342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7291.908461] Lustre: Skipped 1 previous similar message [ 7292.304817] Lustre: server umount lustre-OST0000 complete [ 7302.134338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7302.137879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7302.183601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7303.719404] Lustre: server umount lustre-MDT0000 complete [ 7307.250516] LustreError: 157197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7307.280872] LustreError: 157197:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7308.430081] LustreError: 157179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833115 with bad export cookie 14678343051348309718 [ 7308.453329] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7309.276141] Lustre: server umount lustre-MDT0001 complete [ 7313.892866] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 08:18:37 (1787833117) [ 7316.619751] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7318.555889] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 08:18:42 (1787833122) [ 7328.103996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7328.642659] LustreError: 161570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7328.774283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7333.860630] LustreError: 161571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7334.154305] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7338.985079] LustreError: 161570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7344.102920] LustreError: 161571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7346.203571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7353.450234] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7366.027259] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7373.032793] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7381.073440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7381.589440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7381.594028] Lustre: Skipped 1 previous similar message [ 7388.425457] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7397.703443] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7413.218224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7413.222092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7413.228653] Lustre: Skipped 1 previous similar message [ 7413.243713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7413.248651] Lustre: Skipped 1 previous similar message [ 7416.310575] Lustre: server umount lustre-MDT0000 complete [ 7419.258607] LustreError: 161567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7420.113406] LustreError: 161550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833226 with bad export cookie 14678343051348310600 [ 7420.134261] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7420.135427] LustreError: 161550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7424.425177] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.18@tcp (stopping) [ 7424.434179] Lustre: Skipped 4 previous similar messages [ 7425.009295] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7425.031954] Lustre: Skipped 2 previous similar messages [ 7426.553671] Lustre: server umount lustre-MDT0001 complete [ 7434.061725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7448.427694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7448.695099] LustreError: 165435:0:(llog.c:1655:llog_backup()) MGC192.168.201.118@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7448.705405] Lustre: 165435:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.118@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7449.571021] LustreError: 165440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7449.586084] LustreError: 165440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 7485.420017] LustreError: 165441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7485.445463] LustreError: 165441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [ 7509.472309] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7509.481457] Lustre: 165435:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.118@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7550.457257] LustreError: 165442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7550.468435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7550.482747] LustreError: 165442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 43 previous similar messages [ 7555.585541] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7555.688371] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7567.377363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7581.161439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7590.441937] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcbb3f33c6af093a1 [ 7590.450567] Lustre: MGC192.168.201.118@tcp: Connection restored to 0@lo (at 0@lo) [ 7595.291831] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7596.023768] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7603.400618] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7605.086545] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7611.995507] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7613.973844] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7636.236747] Lustre: server umount lustre-MDT0000 complete [ 7636.968613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7636.977430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7636.994528] Lustre: Skipped 2 previous similar messages [ 7640.406138] LustreError: 161550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833447 with bad export cookie 14678343051348317089 [ 7640.414862] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7640.424600] LustreError: 161550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7640.839203] Lustre: server umount lustre-MDT0001 complete [ 7655.199536] Lustre: server umount lustre-OST0000 complete [ 7681.579897] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 7693.658874] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7732.399913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7732.615634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7732.646571] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7732.724158] Lustre: lustre-MDT0000: new disk, initializing [ 7732.806489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7732.820939] Lustre: Skipped 1 previous similar message [ 7732.842721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7737.735989] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7749.524767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7749.711807] Lustre: 170545:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7750.110195] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7755.586953] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7765.483275] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7771.505322] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7778.948965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7779.273759] Lustre: lustre-OST0000: new disk, initializing [ 7779.276917] Lustre: Skipped 1 previous similar message [ 7779.280746] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7779.285573] Lustre: Skipped 2 previous similar messages [ 7779.290612] Lustre: 172514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7786.959134] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7787.166614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7787.176216] Lustre: Skipped 1 previous similar message [ 7787.181735] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7787.306368] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7798.378540] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7804.001602] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7804.252182] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7809.317346] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7809.559943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7813.090898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7813.093623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7813.117082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7813.131451] Lustre: Skipped 3 previous similar messages [ 7825.889711] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7826.163448] Lustre: server umount lustre-OST0000 complete [ 7832.038168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7837.581627] Lustre: server umount lustre-MDT0000 complete [ 7838.689378] LustreError: 170551:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7838.703083] LustreError: 170551:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 36 previous similar messages [ 7841.000560] LustreError: 170537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833647 with bad export cookie 14678343051348333896 [ 7841.002629] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7841.012344] LustreError: 170537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7841.029689] LustreError: 148622:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [ 7841.258191] Lustre: server umount lustre-MDT0001 complete [ 7849.855925] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 08:27:33 (1787833653) [ 7856.390497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7861.725503] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7870.313441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7874.973204] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7882.610411] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7888.737812] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7896.596470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7904.821476] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7914.290433] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7930.043731] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 08:28:54 (1787833734) [ 7935.975067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7935.979899] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7936.000071] Lustre: Skipped 3 previous similar messages [ 7936.019093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7936.036926] Lustre: Skipped 8 previous similar messages [ 7940.639347] Lustre: server umount lustre-OST0000 complete [ 7951.628928] Lustre: server umount lustre-MDT0000 complete [ 7956.444590] LustreError: 174909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833763 with bad export cookie 14678343051348334778 [ 7956.447730] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7956.461230] LustreError: 174909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7956.840972] Lustre: server umount lustre-MDT0001 complete [ 7969.018449] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 7972.012137] Key type lgssc unregistered [ 7972.221152] LNet: 178662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7972.228503] LNetError: 178662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7972.248562] LNet: Removed LNI 192.168.201.118@tcp [ 7973.289623] Key type .llcrypt unregistered [ 7973.291929] Key type ._llcrypt unregistered [ 7992.503377] Key type ._llcrypt registered [ 7992.505894] Key type .llcrypt registered [ 7992.644137] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7993.834347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7993.865429] alg: No test for adler32 (adler32-zlib) [ 7994.933484] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 7995.151486] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 7996.800368] Key type lgssc registered [ 7998.017491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8006.709437] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8006.730896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8022.368450] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8027.488328] LustreError: 179910:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.118@tcp: failed processing log, type 4: rc = -110 [ 8032.611948] LustreError: 179910:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8032.625855] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8032.633521] LustreError: MGC192.168.201.118@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8032.648950] LustreError: 179910:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8032.664364] LustreError: 179910:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 8032.677517] LustreError: 179910:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8032.776521] Lustre: server umount lustre-OST0000 complete [ 8032.781770] LustreError: 179910:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8041.681235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8043.054705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8047.419852] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8053.230771] LustreError: 180410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8053.252076] LustreError: 180410:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8054.818469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8055.087960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8059.476974] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8066.648385] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8072.414838] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8079.918598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8080.050334] LustreError: 182225:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8080.056536] Lustre: 182225:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.118@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8080.159455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8087.115795] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8094.225107] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8096.245743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8096.259063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8096.270580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8098.788731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8098.791428] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8099.874290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8099.891345] Lustre: Skipped 4 previous similar messages [ 8102.664442] Lustre: server umount lustre-MDT0000 complete [ 8104.937622] LustreError: 180409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8106.608513] LustreError: 180389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833913 with bad export cookie 14109516958293430307 [ 8106.612929] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8106.622507] LustreError: 180389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8108.128345] LustreError: 181137:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000aad15cc0 x1874679615747584/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8108.268787] Lustre: server umount lustre-MDT0001 complete [ 8121.576259] Lustre: server umount lustre-OST0000 complete [ 8132.029979] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 8143.839571] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8184.813527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8185.072276] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8185.098562] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8185.233445] Lustre: lustre-MDT0000: new disk, initializing [ 8185.330085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8185.343528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8190.224093] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8200.681467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8200.834571] Lustre: 185751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8200.876924] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8200.879686] Lustre: Skipped 1 previous similar message [ 8200.952098] Lustre: lustre-MDT0001: new disk, initializing [ 8201.048560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8201.100686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8201.116353] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8205.540526] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8215.925825] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8222.313909] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8229.637087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8230.002711] Lustre: lustre-OST0000: new disk, initializing [ 8230.007970] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8230.013393] Lustre: 187720:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8230.148872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8236.508064] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8237.148513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8237.157036] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8237.308293] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8245.877229] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8251.231547] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8251.455297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8256.512268] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8256.720287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8263.137637] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8263.139334] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8263.154588] LustreError: Skipped 1 previous similar message [ 8263.157730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8263.176476] Lustre: Skipped 2 previous similar messages [ 8263.201621] Lustre: Skipped 1 previous similar message [ 8268.278841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8268.289839] Lustre: Skipped 1 previous similar message [ 8273.376345] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8273.752131] Lustre: server umount lustre-OST0000 complete [ 8277.986738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8278.006227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8278.012962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8278.015724] Lustre: Skipped 2 previous similar messages [ 8283.828991] Lustre: server umount lustre-MDT0000 complete [ 8287.708925] LustreError: 185743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834094 with bad export cookie 14109516958293431028 [ 8287.712017] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8287.729603] LustreError: 185743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8288.038887] Lustre: server umount lustre-MDT0001 complete [ 8297.481609] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 08:35:01 (1787834101) [ 8315.321395] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 8318.345612] Key type lgssc unregistered [ 8318.734046] LNet: 190669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8318.740166] LNetError: 190669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8318.761668] LNet: Removed LNI 192.168.201.118@tcp [ 8319.743878] Key type .llcrypt unregistered [ 8319.745543] Key type ._llcrypt unregistered [ 8342.011703] Key type ._llcrypt registered [ 8342.013948] Key type .llcrypt registered [ 8342.148963] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8343.498488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8343.532737] alg: No test for adler32 (adler32-zlib) [ 8344.670533] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 8344.976562] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 8346.704194] Key type lgssc registered [ 8347.928640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8354.799975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8354.819865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8356.220310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8361.133792] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8365.548633] LustreError: 191889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8365.577535] LustreError: 191889:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8370.098030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8370.509294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8376.107767] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8384.078926] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8391.116045] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8399.867597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8400.094642] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8400.104214] LustreError: 193755:0:(llog.c:1696:llog_backup()) MGC192.168.201.118@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8400.113508] Lustre: 193755:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.118@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8400.264095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8407.311842] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8415.746979] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8420.839089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8420.843607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8420.854186] Lustre: Skipped 1 previous similar message [ 8420.878083] Lustre: Skipped 2 previous similar messages [ 8424.724895] Lustre: server umount lustre-MDT0000 complete [ 8425.956530] LustreError: 191885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8425.973339] LustreError: 191885:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8428.688323] LustreError: 191870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834235 with bad export cookie 12943532981566366639 [ 8428.693450] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8428.712537] LustreError: 191870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8429.135292] Lustre: server umount lustre-MDT0001 complete [ 8444.439155] Lustre: server umount lustre-OST0000 complete [ 8456.542789] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 8467.802464] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8511.684663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8512.038995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8512.059540] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8512.136576] Lustre: lustre-MDT0000: new disk, initializing [ 8512.273211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8512.293837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8517.411920] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8530.919274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8531.158231] Lustre: 197281:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8531.203730] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8531.209752] Lustre: Skipped 1 previous similar message [ 8531.212705] LustreError: 198064:0:(llog.c:1696:llog_backup()) MGC192.168.201.118@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8531.229223] LustreError: 198064:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8531.235193] Lustre: 198064:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.118@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8531.279080] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8531.288236] Lustre: Skipped 1 previous similar message [ 8531.436514] Lustre: lustre-MDT0001: new disk, initializing [ 8531.620397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8531.676323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8531.688043] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8537.813284] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8548.231797] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8555.047804] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8563.501159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8563.838728] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8563.844145] Lustre: Skipped 1 previous similar message [ 8563.850856] LustreError: 199224:0:(llog.c:1696:llog_backup()) MGC192.168.201.118@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8563.868349] LustreError: 199224:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8563.878651] Lustre: 199224:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.118@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8563.900942] Lustre: 199224:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8563.926713] Lustre: lustre-OST0000: new disk, initializing [ 8563.938337] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8563.946845] Lustre: 199249:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8564.044810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8571.616995] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8574.089036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8574.107520] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8574.257634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8582.909298] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8588.754586] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8588.951920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8593.602540] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8593.827634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8600.045745] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8600.052910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8600.055674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8600.073660] Lustre: Skipped 2 previous similar messages [ 8600.088703] Lustre: Skipped 1 previous similar message [ 8605.156022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8605.163868] Lustre: Skipped 1 previous similar message [ 8610.272914] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8610.274309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8610.306449] Lustre: Skipped 1 previous similar message [ 8610.442758] Lustre: server umount lustre-OST0000 complete [ 8615.395414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8615.415282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8615.422188] Lustre: Skipped 1 previous similar message [ 8620.864688] Lustre: server umount lustre-MDT0000 complete [ 8624.968732] LustreError: 199247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834431 with bad export cookie 12943532981566367360 [ 8624.975894] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8624.985081] LustreError: 199247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8625.564865] Lustre: server umount lustre-MDT0001 complete [ 8635.425405] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 08:40:39 (1787834439) [ 8653.490559] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8664.339906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8664.830365] LustreError: 202378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8664.895432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8670.179526] LustreError: 202379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8670.230322] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8674.290945] LustreError: 202378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8679.838691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8685.398445] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8688.300439] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8695.610808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8702.267912] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8711.931715] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8712.125460] Lustre: lustre-OST0001: new disk, initializing [ 8712.132017] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8712.143415] Lustre: 204686:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8712.247518] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8712.260779] Lustre: Skipped 2 previous similar messages [ 8718.144759] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8720.005219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8720.018091] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8720.124621] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8730.712956] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8742.130762] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8744.661866] Lustre: DEBUG MARKER: ADD OST3 [ 8750.157630] loop: module loaded [ 8750.277716] loop0: detected capacity change from 0 to 204800000 [ 8750.713410] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8750.725765] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8750.736923] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8758.819499] Lustre: DEBUG MARKER: STOP OST3 [ 8759.968353] loop0: detected capacity change from 0 to 204800000 [ 8766.493248] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8766.686582] Lustre: lustre-OST0003: new disk, initializing [ 8766.689098] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8766.692807] Lustre: 206771:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8768.311841] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8768.524568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8768.534580] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8768.592312] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8768.878243] Lustre: Failing over lustre-OST0003 [ 8769.011326] Lustre: server umount lustre-OST0003 complete [ 8770.338586] LustreError: 206921:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8773.530280] LustreError: 203864:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8773.555765] LustreError: 203864:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8773.619488] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8773.623252] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8773.626874] LustreError: Skipped 1 previous similar message [ 8773.626914] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8773.626917] Lustre: Skipped 1 previous similar message [ 8784.329601] Lustre: DEBUG MARKER: START OST3 again [ 8791.895251] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8792.340580] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8792.356223] Lustre: Skipped 1 previous similar message [ 8792.370546] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8793.461355] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8793.756909] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8793.757346] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8800.754047] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8822.826994] Lustre: Failing over lustre-OST0003 [ 8823.026428] Lustre: server umount lustre-OST0003 complete [ 8823.289387] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8823.292553] LustreError: 203865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8823.306487] Lustre: Skipped 2 previous similar messages [ 8823.331359] LustreError: 203865:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 8843.748937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8843.776876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8843.779924] Lustre: Skipped 3 previous similar messages [ 8843.802959] Lustre: Skipped 5 previous similar messages [ 8848.049553] Lustre: server umount lustre-MDT0000 complete [ 8851.840843] LustreError: 206770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834658 with bad export cookie 12943532981566368242 [ 8851.860799] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8856.994103] LustreError: 205804:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8857.019277] LustreError: 205804:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 8862.121161] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8862.137719] Lustre: Skipped 5 previous similar messages [ 8866.272392] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8866.564384] Lustre: server umount lustre-MDT0001 complete [ 8870.752606] Lustre: server umount lustre-OST0000 complete [ 8874.700789] Lustre: server umount lustre-OST0001 complete [ 8881.915338] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 8891.191538] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8934.870410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8935.237385] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8935.289474] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8935.403768] Lustre: lustre-MDT0000: new disk, initializing [ 8935.529171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8935.566673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8940.122994] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8951.547788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8951.715735] Lustre: 212298:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8951.771968] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8951.774377] Lustre: Skipped 1 previous similar message [ 8951.878650] Lustre: lustre-MDT0001: new disk, initializing [ 8952.057372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8952.066714] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8956.429945] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8965.621762] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8971.774771] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8979.697176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8980.229482] Lustre: 214269:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8981.504241] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8981.656378] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8988.151072] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8998.285360] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9003.364949] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9003.630399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9008.722330] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9008.950819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9012.195104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9012.204453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9012.215409] Lustre: Skipped 2 previous similar messages [ 9012.218589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9017.157285] Lustre: server umount lustre-OST0000 complete [ 9035.232167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9035.469522] Lustre: server umount lustre-MDT0000 complete [ 9038.309835] LustreError: 213109:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9038.337789] LustreError: 213109:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9040.555758] LustreError: 212290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834847 with bad export cookie 12943532981566370538 [ 9040.561026] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9040.564077] LustreError: 212290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9041.131727] Lustre: server umount lustre-MDT0001 complete [ 9045.975621] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 08:47:29 (1787834849) [ 9064.961758] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 9076.268906] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9118.445329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9118.696426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9118.725596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9118.730693] Lustre: Skipped 1 previous similar message [ 9118.845937] Lustre: lustre-MDT0000: new disk, initializing [ 9118.848427] Lustre: Skipped 1 previous similar message [ 9118.931327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9118.940918] Lustre: Skipped 1 previous similar message [ 9124.365617] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9136.724383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9136.931401] Lustre: 218882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9137.408671] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9142.566252] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9151.671817] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9157.117694] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9164.444865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9164.819622] Lustre: 220851:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9166.182515] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9171.800769] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9181.767968] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9186.930638] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9187.145917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9191.641156] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9191.854110] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9197.025813] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9197.027501] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9197.038331] LustreError: Skipped 2 previous similar messages [ 9197.061967] Lustre: Skipped 3 previous similar messages [ 9197.063279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9197.074973] Lustre: Skipped 8 previous similar messages [ 9199.938416] Lustre: server umount lustre-OST0000 complete [ 9204.193476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9218.016156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9218.184941] Lustre: server umount lustre-MDT0000 complete [ 9221.619927] LustreError: 218894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9221.632149] LustreError: 218894:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9222.671513] LustreError: 218876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835029 with bad export cookie 12943532981566371420 [ 9222.681650] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9222.687595] LustreError: 218876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9223.205652] Lustre: server umount lustre-MDT0001 complete [ 9232.218700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9233.148725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9233.161962] Lustre: Skipped 5 previous similar messages [ 9237.143898] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9246.675738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9251.707677] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9259.147444] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9265.330745] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9273.534074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9282.562157] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9290.684310] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9301.989759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9302.002855] Lustre: Skipped 4 previous similar messages [ 9307.500538] Lustre: server umount lustre-OST0000 complete [ 9317.522364] Lustre: server umount lustre-MDT0000 complete [ 9321.018521] LustreError: 224913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835127 with bad export cookie 12943532981566372302 [ 9321.021247] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9321.037850] LustreError: 224913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9321.594960] Lustre: server umount lustre-MDT0001 complete [ 9433.402095] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 9436.333408] Key type lgssc unregistered [ 9436.741139] LNet: 226735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9436.767951] LNetError: 226735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9436.806981] LNet: Removed LNI 192.168.201.118@tcp [ 9438.278864] Key type .llcrypt unregistered [ 9438.286142] Key type ._llcrypt unregistered [ 9470.427778] Key type ._llcrypt registered [ 9470.429100] Key type .llcrypt registered [ 9470.514823] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 9485.933235] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9487.200933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9487.246953] alg: No test for adler32 (adler32-zlib) [ 9488.337648] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 9488.562681] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 9490.240220] Key type lgssc registered [ 9491.124356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9527.290092] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9527.314481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9528.608312] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9528.651207] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9528.746149] Lustre: lustre-MDT0000: new disk, initializing [ 9528.804638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9528.820726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9533.206462] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9543.599981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9543.759659] Lustre: 230091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9543.869730] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9543.873333] Lustre: Skipped 1 previous similar message [ 9543.959314] Lustre: lustre-MDT0001: new disk, initializing [ 9544.055991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9544.091739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9544.102932] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9548.633501] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9557.715429] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9563.299794] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9570.875066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9571.240448] Lustre: lustre-OST0000: new disk, initializing [ 9571.244646] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9571.253566] Lustre: 232058:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9571.351338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9578.110781] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9580.615156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9580.629312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9586.174790] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9587.432975] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9592.203898] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9592.481232] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9597.541268] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9597.802600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9601.512298] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9601.513260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9601.517438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9601.521099] Lustre: Skipped 1 previous similar message [ 9601.525048] LustreError: Skipped 1 previous similar message [ 9605.897180] Lustre: server umount lustre-OST0000 complete [ 9610.722433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9610.737375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9610.756063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9610.762659] Lustre: Skipped 1 previous similar message [ 9611.749969] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9611.769929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9611.781210] Lustre: Skipped 1 previous similar message [ 9616.500884] Lustre: server umount lustre-MDT0000 complete [ 9620.582967] LustreError: 230085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835427 with bad export cookie 13145920989034428973 [ 9620.585538] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9620.597604] LustreError: 230085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9621.127688] Lustre: server umount lustre-MDT0001 complete [ 9641.504642] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9651.381809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9651.676319] LustreError: 234992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9651.696660] LustreError: 234992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9651.734389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9655.953532] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9656.812380] LustreError: 234993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9661.930331] LustreError: 234992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9663.887133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9664.421186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9668.892911] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9671.788614] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9678.401275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9678.882366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9685.569815] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9694.397791] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9694.546872] Lustre: lustre-OST0001: new disk, initializing [ 9694.552651] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9694.557760] Lustre: 237299:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9701.041724] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9701.997472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9702.005019] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9702.124675] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9713.222993] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9723.320278] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9737.020642] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 08:59:00 (1787835540) [ 9741.283988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9741.300119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9741.317514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9745.902942] Lustre: server umount lustre-MDT0000 complete [ 9748.456121] LustreError: 234992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9748.477610] LustreError: 234992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9750.131515] LustreError: 234974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835556 with bad export cookie 13145920989034429855 [ 9750.137556] LustreError: 234974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9750.141738] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9750.574142] Lustre: server umount lustre-MDT0001 complete [ 9765.153330] Lustre: server umount lustre-OST0000 complete [ 9779.303451] Lustre: server umount lustre-OST0001 complete [ 9785.924611] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 9795.329031] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9834.381938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9834.817324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9834.879972] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9835.055121] Lustre: lustre-MDT0000: new disk, initializing [ 9835.150814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9835.167399] Lustre: Skipped 1 previous similar message [ 9835.187262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9840.790626] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9852.846460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9852.953918] Lustre: 241249:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9852.987785] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9852.991394] Lustre: Skipped 1 previous similar message [ 9853.051185] Lustre: lustre-MDT0001: new disk, initializing [ 9853.147920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9853.166499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9858.877554] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9870.162566] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9876.505674] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9883.242422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9883.551966] Lustre: lustre-OST0000: new disk, initializing [ 9883.556549] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9883.566571] Lustre: 243231:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9883.643921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9883.649333] Lustre: Skipped 1 previous similar message [ 9884.962274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9884.982995] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9885.054646] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9890.928704] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9902.435756] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9909.185903] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9909.479842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9915.025267] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9915.281251] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9920.994845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9920.996478] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9921.014502] LustreError: Skipped 1 previous similar message [ 9921.019123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9921.037423] Lustre: Skipped 4 previous similar messages [ 9921.064038] Lustre: Skipped 5 previous similar messages [ 9923.658945] Lustre: server umount lustre-OST0000 complete [ 9930.215039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9930.225548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9930.248335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9934.046348] Lustre: server umount lustre-MDT0000 complete [ 9936.363355] LustreError: 241261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9938.533502] LustreError: 241240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835745 with bad export cookie 13145920989034431983 [ 9938.564512] LustreError: 241240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9938.567863] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9939.064804] Lustre: server umount lustre-MDT0001 complete [ 9947.989944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9948.463111] LustreError: 245449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9948.481268] LustreError: 245449:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9948.529708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9953.394972] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9962.367778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9968.527086] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9978.392192] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9985.385944] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9993.689441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10000.478665] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10009.475978] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10157.025052] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10157.028183] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10157.039930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10157.046144] Lustre: Skipped 2 previous similar messages [10157.064054] Lustre: Skipped 2 previous similar messages [10162.744775] Lustre: server umount lustre-OST0000 complete [10173.730628] Lustre: server umount lustre-MDT0000 complete [10177.506218] LustreError: 246392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10177.530490] LustreError: 246392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10179.073748] LustreError: 245429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835985 with bad export cookie 13145920989034432872 [10179.079184] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10179.081662] LustreError: 245429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10179.504615] Lustre: server umount lustre-MDT0001 complete [10282.205746] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [10285.511365] Key type lgssc unregistered [10285.856508] LNet: 249061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10285.863129] LNetError: 249061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10285.883298] LNet: Removed LNI 192.168.201.118@tcp [10286.727277] Key type .llcrypt unregistered [10286.732620] Key type ._llcrypt unregistered [10316.386110] Key type ._llcrypt registered [10316.387336] Key type .llcrypt registered [10316.480069] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [10335.295186] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10336.540859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10336.647583] alg: No test for adler32 (adler32-zlib) [10337.980633] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [10338.293919] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [10339.952989] Key type lgssc registered [10341.696406] Lustre: Echo OBD driver; http://www.lustre.org/ [10403.153724] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10414.153483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10414.186606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10415.523253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10415.585481] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10415.704865] Lustre: lustre-MDT0000: new disk, initializing [10415.827253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10415.858607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10420.869945] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10432.985937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10433.135918] Lustre: 253135:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10433.187830] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10433.191826] Lustre: Skipped 1 previous similar message [10433.268374] Lustre: lustre-MDT0001: new disk, initializing [10433.367484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10433.399355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10433.421054] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10438.903542] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10444.116434] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10451.565860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10451.888544] Lustre: lustre-OST0000: new disk, initializing [10451.891707] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10451.900507] Lustre: 254769:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10451.983992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10454.075570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10454.097746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10454.256293] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10459.513810] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10471.661183] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10471.849695] Lustre: lustre-OST0001: new disk, initializing [10471.853658] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10471.865886] Lustre: 255640:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10472.011305] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10479.165402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10479.170862] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10479.271646] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10479.470150] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10492.514849] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10499.032927] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10515.427864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10515.434074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10515.453954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10515.484937] Lustre: Skipped 3 previous similar messages [10519.944155] Lustre: server umount lustre-MDT0000 complete [10524.527304] LustreError: 254768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836331 with bad export cookie 13059648074391462659 [10524.553100] LustreError: 254768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10524.564547] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10525.136596] Lustre: server umount lustre-MDT0001 complete [10540.096367] Lustre: server umount lustre-OST0000 complete [10542.049318] Lustre: 250642:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787836332/real 1787836332] req@ffff8e1c7ac743c0 x1874682073661056/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787836348 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10542.120168] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10542.150900] Lustre: Skipped 3 previous similar messages [10545.040703] Lustre: server umount lustre-OST0001 complete [10562.161661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10574.798843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10585.684495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10597.345485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10618.404975] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10631.106022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10631.129308] Lustre: lustre-MDT0000: reset Object Index mappings [10631.786716] LustreError: 260654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10631.809227] LustreError: 260654:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10631.882496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10636.999453] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10637.306386] LustreError: 260655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10641.445922] LustreError: 260654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10646.503753] LustreError: 260655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10647.825971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10647.843416] Lustre: lustre-MDT0001: reset Object Index mappings [10648.440100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10654.580022] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10658.578319] Lustre: 261799:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10667.120843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10667.655573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10673.767607] LustreError: 262153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10676.333571] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10676.886290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10682.351451] LustreError: 262424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10682.369035] LustreError: 262424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10687.345687] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10693.122210] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10695.765977] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10704.619686] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10708.903496] Lustre: 263671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10715.111159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10715.119671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10715.130765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10716.808492] Lustre: server umount lustre-MDT0000 complete [10718.693208] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10718.695493] LustreError: 260654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10718.713404] Lustre: Skipped 1 previous similar message [10718.740344] LustreError: 260654:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10721.034795] LustreError: 260635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836527 with bad export cookie 13059648074391469449 [10721.037085] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10721.057415] LustreError: 260635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10721.480151] Lustre: server umount lustre-MDT0001 complete [10735.495284] Lustre: server umount lustre-OST0000 complete [10739.362391] Lustre: 250642:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787836530/real 1787836530] req@ffff8e1b4e759680 x1874682073713024/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787836546 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10739.397992] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10739.408996] Lustre: Skipped 1 previous similar message [10740.034585] Lustre: server umount lustre-OST0001 complete [10762.598433] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 09:16:06 (1787836566) [10782.658394] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [10794.245644] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10837.835747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10838.250392] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10838.278640] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10838.330226] Lustre: lustre-MDT0000: new disk, initializing [10838.439374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10838.443758] Lustre: Skipped 1 previous similar message [10838.453464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10843.585512] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10855.457900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10855.602759] Lustre: 267928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10855.643594] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10855.646759] Lustre: Skipped 1 previous similar message [10855.734646] Lustre: lustre-MDT0001: new disk, initializing [10855.862372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10855.884858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10862.467430] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10874.908996] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10882.263509] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10889.402832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10889.714479] Lustre: lustre-OST0000: new disk, initializing [10889.718337] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10889.727381] Lustre: 269892:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10891.240198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10891.257487] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10891.415637] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10897.621826] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10908.023708] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10913.375987] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10913.588919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10918.403825] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10918.597753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10921.963159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10921.968975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10921.973984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10921.989019] Lustre: Skipped 1 previous similar message [10922.002662] Lustre: Skipped 1 previous similar message [10927.077139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10927.083578] Lustre: Skipped 1 previous similar message [10932.199354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10932.215174] Lustre: Skipped 1 previous similar message [10935.265523] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10935.477897] Lustre: server umount lustre-OST0000 complete [10942.433241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10942.458221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10947.554500] LustreError: 271233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10947.575857] LustreError: 271233:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10947.633290] Lustre: server umount lustre-MDT0000 complete [10952.053746] LustreError: 267918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836758 with bad export cookie 13059648074391471542 [10952.060465] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10952.065455] LustreError: 267918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10952.442365] Lustre: server umount lustre-MDT0001 complete [10972.723479] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10983.093418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10983.576164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10983.584732] Lustre: Skipped 2 previous similar messages [10988.145784] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10996.980629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11001.882695] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11005.430272] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11013.191459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11021.036257] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11030.264159] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11030.480737] Lustre: lustre-OST0001: new disk, initializing [11030.489345] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11030.501948] Lustre: 275142:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11032.380894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11032.393823] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11032.502320] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11037.547249] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11046.919797] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11050.210440] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11055.230857] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11055.307615] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11063.268822] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11063.284112] Lustre: Skipped 1 previous similar message [11063.286796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11063.299050] Lustre: Skipped 4 previous similar messages [11068.083514] Lustre: server umount lustre-MDT0000 complete [11068.385768] LustreError: 272833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11068.404983] LustreError: 272833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [11071.427020] LustreError: 272813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836878 with bad export cookie 13059648074391472424 [11071.449442] LustreError: 272813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11071.452937] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11071.749537] Lustre: server umount lustre-MDT0001 complete [11086.399960] Lustre: server umount lustre-OST0000 complete [11089.635139] Lustre: 250643:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787836880/real 1787836880] req@ffff8e1b51b7fc00 x1874682073829120/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787836896 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11090.372358] Lustre: server umount lustre-OST0001 complete [11107.994260] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [11119.598874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11119.982217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11119.984888] Lustre: Skipped 3 previous similar messages [11124.595938] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11134.030115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11138.892377] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11142.586420] Lustre: 279337:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11149.927625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11157.977230] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11164.910082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11167.964606] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11173.374956] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11175.228480] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11184.055990] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11187.996925] Lustre: 281214:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11193.635675] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11193.669114] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11204.257650] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 09:23:27 (1787837007) [11219.427644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11219.436380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11219.444878] Lustre: Skipped 6 previous similar messages [11219.450565] Lustre: Skipped 1 previous similar message [11225.813664] Lustre: server umount lustre-MDT0000 complete [11229.077477] LustreError: 278177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837035 with bad export cookie 13059648074391479186 [11229.082490] LustreError: 278177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11229.087859] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11229.665528] LustreError: 280978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11229.692668] LustreError: 280978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [11235.901691] Lustre: server umount lustre-MDT0001 complete [11246.450991] Lustre: server umount lustre-OST0000 complete [11251.031595] Lustre: server umount lustre-OST0001 complete [11258.379859] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [11268.093457] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [11316.257164] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [11327.163538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11327.378336] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11327.398976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11327.471566] Lustre: lustre-MDT0000: new disk, initializing [11327.559484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11332.260842] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11343.790741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11343.936285] Lustre: 285458:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11344.271194] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11349.378191] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11361.683053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11361.952571] Lustre: 287088:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11363.378615] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11369.611936] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11380.171101] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11380.340289] Lustre: 287961:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11380.438024] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11380.442231] Lustre: Skipped 6 previous similar messages [11381.806305] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11381.813927] Lustre: Skipped 1 previous similar message [11381.886366] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11386.121775] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11395.650825] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11399.199847] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11399.204960] Lustre: Skipped 1 previous similar message [11414.078661] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 09:26:58 (1787837218) [11417.581714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11417.585124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11417.594967] Lustre: Skipped 6 previous similar messages [11417.609429] Lustre: Skipped 10 previous similar messages [11418.673201] Lustre: server umount lustre-MDT0000 complete [11422.356657] LustreError: 287960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837229 with bad export cookie 13059648074391486599 [11422.369782] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11422.378644] LustreError: 287960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11422.691134] Lustre: server umount lustre-MDT0001 complete [11435.540309] Lustre: server umount lustre-OST0000 complete [11438.497332] Lustre: 250640:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787837229/real 1787837229] req@ffff8e1c67d45a40 x1874682073978752/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787837245 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11439.365961] Lustre: server umount lustre-OST0001 complete [11447.941914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11452.806584] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11461.060318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11465.994825] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11474.296262] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11481.103540] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11488.531289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11494.247672] LustreError: 292033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11494.278818] LustreError: 292033:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11494.680088] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11501.729958] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11522.809740] loop1: detected capacity change from 0 to 204800000 [11523.063728] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11523.080536] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11523.096117] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11528.997140] loop1: detected capacity change from 0 to 204800000 [11534.143561] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11534.232394] LustreError: 290196:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs6361: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11534.256296] LustreError: 290196:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs6361 [11534.263498] LustreError: 290196:0:(mgs_handler.c:517:mgs_target_reg()) fs6361-MDT0001: Error checking target index: rc = -13 [11534.273042] LustreError: fs6361-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11534.281172] LustreError: 293926:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11534.293979] LustreError: 293926:0:(tgt_mount.c:2129:server_put_super()) no obd fs6361-MDT0001 [11534.300141] LustreError: 293926:0:(tgt_mount.c:116:server_deregister_mount()) fs6361-MDT0001 not registered [11534.361643] Lustre: server umount fs6361-MDT0001 complete [11534.366394] LustreError: 293926:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11543.602826] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11543.669486] Lustre: Setting parameter fs6361-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs6361-MDT0001 [11543.721926] Lustre: srv-fs6361-MDT0001: No data found on store. Initialize space. [11543.727428] Lustre: Skipped 4 previous similar messages [11543.789293] Lustre: fs6361-MDT0001: new disk, initializing [11543.793260] Lustre: Skipped 3 previous similar messages [11543.892749] Lustre: srv-fs6361-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11543.900320] Lustre: 294392:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs6361-MDT0001-mdtlov: can't create new llog: rc = -115 [11543.906149] LustreError: 294392:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs6361-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11547.377473] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11553.229318] Lustre: server umount fs6361-MDT0001 complete [11565.398694] loop1: detected capacity change from 0 to 204800000 [11570.928159] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11571.300183] Lustre: 295960:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs6361-MDT0001-mdtlov: can't create new llog: rc = -115 [11571.304471] LustreError: 295960:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs6361-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11573.246212] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11577.234330] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11582.979772] Lustre: server umount fs6361-MDT0001 complete [11588.585279] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11594.634361] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.18@tcp (stopping) [11594.726438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11600.679053] Lustre: server umount lustre-MDT0000 complete [11610.867148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11611.055314] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11616.243326] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11616.773859] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11623.847118] loop1: detected capacity change from 0 to 204800000 [11628.995181] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11629.509986] Lustre: 298409:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs6361-MDT0001-mdtlov: can't create new llog: rc = -115 [11629.519281] LustreError: 298409:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs6361-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11633.902762] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11637.545887] Lustre: server umount fs6361-MDT0001 complete [11657.699734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11671.008599] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11671.166468] Lustre: server umount lustre-OST0000 complete [11678.183077] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11678.194827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11678.198957] Lustre: Skipped 7 previous similar messages [11681.186801] Lustre: server umount lustre-MDT0000 complete [11685.543032] LustreError: 290187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837492 with bad export cookie 13059648074391489280 [11685.551613] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11685.556481] LustreError: 290187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11686.059624] Lustre: server umount lustre-MDT0001 complete [11697.100495] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [11699.456398] Key type lgssc unregistered [11699.694452] LNet: 300757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11699.699956] LNetError: 300757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11699.713295] LNet: Removed LNI 192.168.201.118@tcp [11700.402167] Key type .llcrypt unregistered [11700.403995] Key type ._llcrypt unregistered [11722.549776] Key type ._llcrypt registered [11722.552707] Key type .llcrypt registered [11722.708673] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [11723.805821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11723.822790] alg: No test for adler32 (adler32-zlib) [11725.105664] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [11725.427274] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [11727.088660] Key type lgssc registered [11728.351290] Lustre: Echo OBD driver; http://www.lustre.org/ [11736.489585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11736.508317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11738.161728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11742.274426] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11747.300812] LustreError: 302028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11747.317634] LustreError: 302028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11750.021529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11750.485405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11753.837584] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11765.984802] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11773.857805] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11780.443107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11780.804258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11782.917510] LustreError: 303857:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11785.511226] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11788.258275] LustreError: 303859:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11792.693277] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11793.391358] LustreError: 303991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11800.451267] LustreError: 303857:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11800.464784] LustreError: 303857:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11810.685193] LustreError: 303991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11810.710697] LustreError: 303991:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11814.622141] loop1: detected capacity change from 0 to 204800000 [11814.795409] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11814.825843] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11814.839118] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11819.969535] loop1: detected capacity change from 0 to 204800000 [11825.656272] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11825.801529] Lustre: fs23460-OST0000: new disk, initializing [11825.808361] Lustre: srv-fs23460-OST0000: No data found on store. Initialize space. [11825.826362] Lustre: 305762:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11825.907440] Lustre: fs23460-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11831.163593] LustreError: 303858:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11831.186152] LustreError: 303858:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [11831.827408] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11841.435502] Lustre: server umount fs23460-OST0000 complete [11854.286426] loop1: detected capacity change from 0 to 204800000 [11860.308205] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11860.455150] Lustre: fs23460-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11862.014728] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11866.781119] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11867.013104] LustreError: 303991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11867.042101] LustreError: 303991:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [11876.797439] loop2: detected capacity change from 0 to 204800000 [11877.093996] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11877.103356] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11877.125393] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11877.364332] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11883.263271] loop2: detected capacity change from 0 to 204800000 [11890.290691] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11890.429468] LustreError: fs23460-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11890.436751] LustreError: fs23460-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11890.442822] LustreError: 308806:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11890.454747] LustreError: 308806:0:(tgt_mount.c:2129:server_put_super()) no obd fs23460-OST0001 [11890.461630] LustreError: 308806:0:(tgt_mount.c:116:server_deregister_mount()) fs23460-OST0001 not registered [11890.517116] Lustre: server umount fs23460-OST0001 complete [11890.522488] LustreError: 308806:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11902.893662] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11903.120424] Lustre: fs23460-OST0001: new disk, initializing [11903.125139] Lustre: srv-fs23460-OST0001: No data found on store. Initialize space. [11903.134324] Lustre: 309357:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11903.287488] Lustre: fs23460-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11911.074459] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11920.606583] Lustre: server umount fs23460-OST0000 complete [11930.805328] Lustre: server umount fs23460-OST0001 complete [11933.555279] LustreError: 303991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11933.569925] LustreError: 303991:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11945.784277] loop1: detected capacity change from 0 to 204800000 [11952.316042] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11952.720379] Lustre: fs23460-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11960.205585] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11968.144897] loop2: detected capacity change from 0 to 204800000 [11975.078200] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11982.609772] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11995.617550] Lustre: server umount fs23460-OST0000 complete [12007.101104] Lustre: server umount fs23460-OST0001 complete [12015.605499] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12020.715903] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12020.716368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12020.726555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12020.730369] Lustre: Skipped 1 previous similar message [12020.752883] Lustre: Skipped 1 previous similar message [12025.832773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12026.989982] Lustre: server umount lustre-OST0000 complete [12036.072981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12036.086807] Lustre: Skipped 1 previous similar message [12036.089925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12036.098630] Lustre: Skipped 2 previous similar messages [12038.759440] Lustre: server umount lustre-MDT0000 complete [12043.761402] LustreError: 302010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837850 with bad export cookie 7209785768477065112 [12043.766161] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12043.770404] LustreError: 302010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12044.452230] Lustre: server umount lustre-MDT0001 complete [12059.660910] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12064.817412] Key type lgssc unregistered [12065.285521] LNet: 315285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12065.317088] LNetError: 315285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12065.346413] LNet: Removed LNI 192.168.201.118@tcp [12066.525255] Key type .llcrypt unregistered [12066.526815] Key type ._llcrypt unregistered [12107.930279] Key type ._llcrypt registered [12107.940893] Key type .llcrypt registered [12108.097870] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [12127.386184] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [12129.447541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12129.567661] alg: No test for adler32 (adler32-zlib) [12130.704920] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [12131.014927] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [12132.712765] Key type lgssc registered [12134.186968] Lustre: Echo OBD driver; http://www.lustre.org/ [12177.586540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12177.612916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12178.850881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12178.879699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12178.967664] Lustre: lustre-MDT0000: new disk, initializing [12179.050565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12179.068598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12185.107149] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12197.269603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12197.409818] Lustre: 319147:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12197.483864] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12197.489093] Lustre: Skipped 1 previous similar message [12197.679789] Lustre: lustre-MDT0001: new disk, initializing [12197.842813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12197.888215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12197.899854] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12203.178210] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12214.798374] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12222.267892] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12230.416832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12230.825409] Lustre: lustre-OST0000: new disk, initializing [12230.833546] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12230.843272] Lustre: 321120:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12230.995062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12236.388472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12236.411721] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12236.559557] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12238.581532] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12249.907402] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12252.961504] LustreError: 317319:0:(lmv_obd.c:211:lmv_notify()) activation of lustre-MDT0000_UUID failed: -22 [12253.024496] Lustre: Mounted lustre-client - version 2.17.57_68_geb35bb6 [12254.306246] Lustre: Unmounted lustre-client [12267.489824] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12267.498860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12267.532265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12267.539347] Lustre: Skipped 1 previous similar message [12268.515032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12268.531069] Lustre: Skipped 1 previous similar message [12270.859967] Lustre: server umount lustre-OST0000 complete [12278.758868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12278.781707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12278.785490] Lustre: Skipped 2 previous similar messages [12278.812750] Lustre: Skipped 1 previous similar message [12281.878669] Lustre: server umount lustre-MDT0000 complete [12286.531800] LustreError: 319141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838093 with bad export cookie 4155988432012116104 [12286.554321] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12287.190341] Lustre: server umount lustre-MDT0001 complete [12300.430779] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12303.854852] Key type lgssc unregistered [12313.315068] LNet: 1 local NIs in recovery (showing 1): 192.168.201.118@tcp [12313.443358] LNet: 323282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12313.473348] LNetError: 323282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12384.171310] LNet: Removed LNI 192.168.201.118@tcp [12385.432150] Key type .llcrypt unregistered [12385.438859] Key type ._llcrypt unregistered [12408.746485] Key type ._llcrypt registered [12408.748806] Key type .llcrypt registered [12408.899728] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [12427.164462] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [12428.304542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12428.338374] alg: No test for adler32 (adler32-zlib) [12429.794199] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [12430.133990] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [12431.832154] Key type lgssc registered [12432.924425] Lustre: Echo OBD driver; http://www.lustre.org/ [12468.945281] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 09:44:32 (1787838272) [12476.766238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12476.803085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12478.294658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12478.344947] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12478.500605] Lustre: lustre-MDT0000: new disk, initializing [12478.692058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12478.751110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12484.178114] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12495.536924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12495.685796] Lustre: 326510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12495.792434] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12495.797223] Lustre: Skipped 1 previous similar message [12495.907781] Lustre: lustre-MDT0001: new disk, initializing [12496.095476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12496.129767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12496.147405] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12501.779885] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12512.214955] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12519.795768] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12527.599676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12528.101504] Lustre: lustre-OST0000: new disk, initializing [12528.105436] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12528.111946] Lustre: 328479:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12528.253246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12535.384263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12535.398446] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12535.538224] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12535.633180] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12546.839561] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12571.617878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12571.620495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12571.621555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12571.641660] Lustre: Skipped 1 previous similar message [12571.650045] Lustre: Skipped 1 previous similar message [12572.913391] Lustre: server umount lustre-OST0000 complete [12578.291486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12578.301260] LustreError: Skipped 1 previous similar message [12578.306612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12578.321360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12581.858537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12581.861075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12583.707822] Lustre: server umount lustre-MDT0000 complete [12588.161155] LustreError: 326499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838394 with bad export cookie 15980661560210723612 [12588.167101] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12588.174360] LustreError: 326499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12588.701857] Lustre: server umount lustre-MDT0001 complete [12601.671195] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12604.874386] Key type lgssc unregistered [12605.128744] LNet: 330392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12605.140279] LNetError: 330392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12605.155896] LNet: Removed LNI 192.168.201.118@tcp [12606.018293] Key type .llcrypt unregistered [12606.021440] Key type ._llcrypt unregistered [12630.338780] Key type ._llcrypt registered [12630.342311] Key type .llcrypt registered [12630.487677] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [12631.678100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12631.959822] alg: No test for adler32 (adler32-zlib) [12633.081471] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [12633.354362] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [12635.032181] Key type lgssc registered [12636.300637] Lustre: Echo OBD driver; http://www.lustre.org/ [12645.733527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12645.751409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12647.360535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12652.977304] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12657.635971] LustreError: 331677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12657.648312] LustreError: 331677:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12662.756728] LustreError: 331678:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12663.212511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12663.769721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12668.988219] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12677.954891] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12685.620147] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12694.368331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12694.783246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12701.611480] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12713.034367] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12730.151319] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12732.062151] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12738.027324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12738.029053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12738.055118] Lustre: Skipped 2 previous similar messages [12738.065195] Lustre: Skipped 1 previous similar message [12740.005869] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.18@tcp (stopping) [12742.832547] Lustre: server umount lustre-MDT0000 complete [12743.147135] LustreError: 331672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12743.180281] LustreError: 331672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12747.699412] LustreError: 331657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838554 with bad export cookie 3071738695169858941 [12747.729153] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12747.730851] LustreError: 331657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12748.258881] LustreError: 331672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12748.259101] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12748.268574] LustreError: 331672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12748.297783] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12752.290288] LustreError: 331672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12752.300379] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12752.308135] LustreError: 331672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12752.321409] Lustre: Skipped 1 previous similar message [12757.444578] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12757.451908] Lustre: Skipped 1 previous similar message [12760.431378] LustreError: 331674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12760.447062] LustreError: 331674:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12762.082769] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12762.412777] Lustre: server umount lustre-MDT0001 complete [12766.605432] Lustre: server umount lustre-OST0000 complete [12776.717122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12781.065083] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12784.469692] Lustre: 335882:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.118@tcp [12786.932837] Lustre: server umount MGS complete [12797.320954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12797.810375] LustreError: 336455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12797.852813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12804.370099] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12813.582625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12813.959611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12817.602973] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12826.990598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12827.280403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12833.946408] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12841.736837] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12843.850611] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12849.951842] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12851.758639] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12857.835930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12857.842929] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12857.858979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12857.863964] Lustre: Skipped 1 previous similar message [12858.862341] Lustre: server umount lustre-OST0000 complete [12863.457508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12863.474450] Lustre: Skipped 2 previous similar messages [12869.558655] Lustre: server umount lustre-MDT0000 complete [12873.364513] LustreError: 336436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838680 with bad export cookie 3071738695169859781 [12873.367371] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12873.379038] LustreError: 336436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12873.765334] Lustre: server umount lustre-MDT0001 complete [12887.024525] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12889.825569] Key type lgssc unregistered [12890.091739] LNet: 339894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12890.108988] LNetError: 339894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12891.178905] LNet: Removed LNI 192.168.201.118@tcp [12892.151194] Key type .llcrypt unregistered [12892.152894] Key type ._llcrypt unregistered [12912.602541] Key type ._llcrypt registered [12912.605265] Key type .llcrypt registered [12912.697708] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [12930.452840] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [12931.740967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12931.940385] alg: No test for adler32 (adler32-zlib) [12933.281660] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [12933.629554] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [12935.408190] Key type lgssc registered [12936.482359] Lustre: Echo OBD driver; http://www.lustre.org/ [12973.191741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12973.212135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12974.528289] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12974.575792] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12974.661853] Lustre: lustre-MDT0000: new disk, initializing [12974.744046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12974.779415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12979.376701] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12990.188294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12990.310684] Lustre: 343054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12990.445962] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12990.455431] Lustre: Skipped 1 previous similar message [12990.603809] Lustre: lustre-MDT0001: new disk, initializing [12990.712838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12990.752228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12990.772742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12995.855579] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13005.855785] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13012.249568] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13019.999035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13020.353128] Lustre: lustre-OST0000: new disk, initializing [13020.358174] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13020.364373] Lustre: 345023:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13020.501135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13028.452067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13028.472239] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13028.611766] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13028.785968] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13039.885747] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13044.779654] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13045.075105] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13050.209277] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13050.516269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13054.444611] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13054.458389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13054.498779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13058.759875] Lustre: server umount lustre-OST0000 complete [13064.674952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13064.681652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13064.690454] Lustre: Skipped 2 previous similar messages [13064.717064] Lustre: Skipped 2 previous similar messages [13069.421898] Lustre: server umount lustre-MDT0000 complete [13073.471814] LustreError: 343049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838880 with bad export cookie 7061513640370757936 [13073.472642] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13073.484278] LustreError: 343049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13073.937148] Lustre: server umount lustre-MDT0001 complete [13083.376245] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 09:54:47 (1787838887) [13100.996911] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [13113.708590] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [13157.285177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13157.573646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13157.613910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13157.712867] Lustre: lustre-MDT0000: new disk, initializing [13157.802363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13157.817684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13163.246293] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13177.035797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13177.239532] Lustre: 349785:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13177.411545] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13177.419800] Lustre: Skipped 1 previous similar message [13177.650712] Lustre: lustre-MDT0001: new disk, initializing [13177.869787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13177.964472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13177.986550] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13184.469243] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13196.162594] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13203.112062] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13212.625958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13212.932471] Lustre: lustre-OST0000: new disk, initializing [13212.936814] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13212.944435] Lustre: 351755:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13213.036408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13219.996111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13220.012127] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13220.193166] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13221.036567] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13236.568392] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13281.770780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13281.774085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13281.783047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13281.786088] LustreError: Skipped 1 previous similar message [13281.813630] Lustre: Skipped 1 previous similar message [13286.887082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13286.893731] Lustre: Skipped 2 previous similar messages [13287.206395] Lustre: server umount lustre-OST0000 complete [13295.585834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13295.591288] LustreError: Skipped 1 previous similar message [13295.596775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13295.615765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13299.007194] Lustre: server umount lustre-MDT0000 complete [13302.248639] LustreError: 349793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13302.284142] LustreError: 349793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13303.125859] LustreError: 349776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839109 with bad export cookie 7061513640370758818 [13303.150837] LustreError: 349776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13303.162106] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13303.686112] Lustre: server umount lustre-MDT0001 complete [13317.181691] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13320.669628] Key type lgssc unregistered [13321.094330] LNet: 353935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13321.103043] LNetError: 353935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13322.156952] LNet: Removed LNI 192.168.201.118@tcp [13323.333557] Key type .llcrypt unregistered [13323.336239] Key type ._llcrypt unregistered [13421.896263] Key type ._llcrypt registered [13421.900928] Key type .llcrypt registered [13422.071337] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13422.394318] Key type .llcrypt unregistered [13422.397336] Key type ._llcrypt unregistered [13500.140377] Key type ._llcrypt registered [13500.146549] Key type .llcrypt registered [13500.292921] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13500.688379] Key type .llcrypt unregistered [13500.690793] Key type ._llcrypt unregistered [13533.693613] Key type ._llcrypt registered [13533.696377] Key type .llcrypt registered [13533.828613] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13534.291389] Key type .llcrypt unregistered [13534.293384] Key type ._llcrypt unregistered [13576.198610] Key type ._llcrypt registered [13576.206175] Key type .llcrypt registered [13576.457054] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13577.037353] Key type .llcrypt unregistered [13577.041034] Key type ._llcrypt unregistered [13660.666367] Key type ._llcrypt registered [13660.675190] Key type .llcrypt registered [13660.756210] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13660.964371] Key type .llcrypt unregistered [13660.966651] Key type ._llcrypt unregistered [13695.213323] Key type ._llcrypt registered [13695.216716] Key type .llcrypt registered [13695.338913] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13695.905595] Key type .llcrypt unregistered [13695.906877] Key type ._llcrypt unregistered [13731.874201] Key type ._llcrypt registered [13731.876904] Key type .llcrypt registered [13732.019717] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13732.478746] Key type .llcrypt unregistered [13732.484368] Key type ._llcrypt unregistered [13767.529928] Key type ._llcrypt registered [13767.532583] Key type .llcrypt registered [13767.689788] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13768.189362] Key type .llcrypt unregistered [13768.191719] Key type ._llcrypt unregistered [13801.173400] Key type ._llcrypt registered [13801.178473] Key type .llcrypt registered [13801.353506] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13801.895281] Key type .llcrypt unregistered [13801.899385] Key type ._llcrypt unregistered [13843.214035] Key type ._llcrypt registered [13843.215990] Key type .llcrypt registered [13843.315043] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13843.664369] Key type .llcrypt unregistered [13843.666052] Key type ._llcrypt unregistered [13913.301248] Key type ._llcrypt registered [13913.304818] Key type .llcrypt registered [13913.524208] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13914.094045] Key type .llcrypt unregistered [13914.095996] Key type ._llcrypt unregistered [13947.853414] Key type ._llcrypt registered [13947.857438] Key type .llcrypt registered [13948.102713] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13948.812916] Key type .llcrypt unregistered [13948.846177] Key type ._llcrypt unregistered [13991.819082] Key type ._llcrypt registered [13991.820936] Key type .llcrypt registered [13992.140368] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13993.112469] Key type .llcrypt unregistered [13993.116172] Key type ._llcrypt unregistered [14056.462628] Key type ._llcrypt registered [14056.467059] Key type .llcrypt registered [14056.593951] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [14073.309114] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [14075.126349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14075.169904] alg: No test for adler32 (adler32-zlib) [14076.749345] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [14077.151803] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [14078.981068] Key type lgssc registered [14080.274872] Lustre: Echo OBD driver; http://www.lustre.org/ [14121.498981] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14121.534284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14122.888306] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14122.958234] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14123.020154] Lustre: tst250fs-MDT0000: new disk, initializing [14123.078808] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14123.098494] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14129.144387] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [14142.076696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14142.297943] Lustre: 374381:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14142.360566] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14142.363822] Lustre: Skipped 1 previous similar message [14142.494747] Lustre: tst250fs-MDT0001: new disk, initializing [14142.636445] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14142.728753] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14142.743420] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14148.799456] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [14154.832766] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14162.813227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14163.356764] Lustre: tst250fs-OST0000: new disk, initializing [14163.377031] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14163.390915] Lustre: 376018:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14163.629691] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14167.210319] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14167.238544] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14167.465076] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14172.232883] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [14185.959053] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14186.286366] Lustre: tst250fs-OST0001: new disk, initializing [14186.290938] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14186.296769] Lustre: 376891:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14186.438990] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14192.301029] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14192.308905] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14192.450773] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14195.114917] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [14201.687241] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14204.501522] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [14204.506332] Lustre: Skipped 2 previous similar messages [14224.353600] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14224.360674] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14224.389313] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14228.450471] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14228.462140] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14228.471906] Lustre: Skipped 1 previous similar message [14228.480587] Lustre: Skipped 2 previous similar messages [14230.556987] Lustre: server umount tst250fs-MDT0000 complete [14235.657072] LustreError: 376017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840042 with bad export cookie 7750198919159522843 [14235.663377] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14235.666445] LustreError: 376017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14236.186859] Lustre: server umount tst250fs-MDT0001 complete [14251.081477] Lustre: server umount tst250fs-OST0000 complete [14255.080808] Lustre: 372603:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787840045/real 1787840045] req@ffff8e1b47850780 x1874685993229568/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787840061 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14255.100329] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14255.108829] Lustre: Skipped 1 previous similar message [14255.772847] Lustre: server umount tst250fs-OST0001 complete [14278.838867] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [14287.664806] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [14323.727750] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 10:15:27 (1787840127) [14325.645366] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14343.017947] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [14353.748731] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [14388.114772] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14070 sec ========= 10:16:31 (1787840191) [14390.536911] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:16:33 (1787840193) === [14394.471192] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:16:38 (1787840198) === [14444.689376] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [14448.416266] Key type lgssc unregistered [14448.669145] LNet: 386025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14448.676080] LNetError: 386025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14448.694555] LNet: Removed LNI 192.168.201.118@tcp [14449.597195] Key type .llcrypt unregistered [14449.600201] Key type ._llcrypt unregistered