[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 546430955 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002410] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007992] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.011062] LSM: Security Framework initializing [ 0.012026] Yama: becoming mindful. [ 0.012729] SELinux: Initializing. [ 0.013037] *** VALIDATE selinux *** [ 0.020570] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025428] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026251] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027141] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029026] *** VALIDATE tmpfs *** [ 0.031151] *** VALIDATE proc *** [ 0.032333] *** VALIDATE cgroup *** [ 0.033017] *** VALIDATE cgroup2 *** [ 0.035130] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037042] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038025] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040051] Spectre V2 : User space: Vulnerable [ 0.041016] Speculative Store Bypass: Vulnerable [ 0.045513] debug: unmapping init [mem 0xffffffffafa59000-0xffffffffafa60fff] [ 0.047283] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048985] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050037] ... version: 2 [ 0.051018] ... bit width: 48 [ 0.052014] ... generic registers: 4 [ 0.053013] ... value mask: 0000ffffffffffff [ 0.054014] ... max period: 00007fffffffffff [ 0.055015] ... fixed-purpose events: 3 [ 0.056022] ... event mask: 000000070000000f [ 0.057338] rcu: Hierarchical SRCU implementation. [ 0.059577] smp: Bringing up secondary CPUs ... [ 0.060649] x86: Booting SMP configuration: [ 0.061029] .... node #0, CPUs: #1 #2 #3 [ 0.066150] smp: Brought up 1 node, 4 CPUs [ 0.068018] smpboot: Max logical packages: 1 [ 0.069011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.153026] node 0 deferred pages initialised in 81ms [ 0.161108] devtmpfs: initialized [ 0.162360] x86/mm: Memory block size: 128MB [ 0.166000] gcov: version magic: 0x41383552 [ 0.170408] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.177167] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.183127] pinctrl core: initialized pinctrl subsystem [ 0.186255] [ 0.187000] ************************************************************* [ 0.190021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.193015] ** ** [ 0.197014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.202025] ** ** [ 0.206012] ** This means that this kernel is built to expose internal ** [ 0.210014] ** IOMMU data structures, which may compromise security on ** [ 0.212010] ** your system. ** [ 0.216014] ** ** [ 0.219023] ** If you see this message and you are not debugging the ** [ 0.222029] ** kernel, report this immediately to your vendor! ** [ 0.226020] ** ** [ 0.228013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.231012] ************************************************************* [ 0.235564] NET: Registered protocol family 16 [ 0.238268] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.242064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.246079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.250115] cpuidle: using governor menu [ 0.254372] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.257414] PCI: Using configuration type 1 for base access [ 0.259138] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.269053] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.271052] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.276091] cryptd: max_cpu_qlen set to 1000 [ 0.279425] ACPI: Added _OSI(Module Device) [ 0.280010] ACPI: Added _OSI(Processor Device) [ 0.282011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.283010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.289000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.296014] ACPI: Interpreter enabled [ 0.297107] ACPI: PM: (supports S0 S3 S4 S5) [ 0.299009] ACPI: Using IOAPIC for interrupt routing [ 0.300104] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.301376] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.312548] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.314030] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.317016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.320080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.325683] acpiphp: Slot [2] registered [ 0.328119] acpiphp: Slot [5] registered [ 0.329107] acpiphp: Slot [6] registered [ 0.331141] acpiphp: Slot [7] registered [ 0.332091] acpiphp: Slot [8] registered [ 0.334279] acpiphp: Slot [9] registered [ 0.336147] acpiphp: Slot [10] registered [ 0.337108] acpiphp: Slot [3] registered [ 0.339104] acpiphp: Slot [4] registered [ 0.341091] acpiphp: Slot [11] registered [ 0.342073] acpiphp: Slot [12] registered [ 0.344114] acpiphp: Slot [13] registered [ 0.346084] acpiphp: Slot [14] registered [ 0.348104] acpiphp: Slot [15] registered [ 0.349136] acpiphp: Slot [16] registered [ 0.351089] acpiphp: Slot [17] registered [ 0.352157] acpiphp: Slot [18] registered [ 0.354087] acpiphp: Slot [19] registered [ 0.356117] acpiphp: Slot [20] registered [ 0.358102] acpiphp: Slot [21] registered [ 0.360147] acpiphp: Slot [22] registered [ 0.361079] acpiphp: Slot [23] registered [ 0.363090] acpiphp: Slot [24] registered [ 0.365119] acpiphp: Slot [25] registered [ 0.367168] acpiphp: Slot [26] registered [ 0.369093] acpiphp: Slot [27] registered [ 0.371138] acpiphp: Slot [28] registered [ 0.372139] acpiphp: Slot [29] registered [ 0.374159] acpiphp: Slot [30] registered [ 0.376143] acpiphp: Slot [31] registered [ 0.378104] PCI host bridge to bus 0000:00 [ 0.380026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.384035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.388031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.391039] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.395029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.398021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.401296] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.404371] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.408840] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.423929] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.429056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.432030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.435017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.438023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.441400] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.443845] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.448054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.452022] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.456017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.467014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.471018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.475885] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.482016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.488019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.505016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.522587] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.537028] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.552106] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.584016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.599035] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.608015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.615014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.629017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.635000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.643017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.648016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.667014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.682934] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.694025] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.704032] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.727021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.744958] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.764020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.776015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.792021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.804098] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.806335] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.810418] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.813455] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.816259] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.824080] iommu: Default domain type: Passthrough [ 0.825000] SCSI subsystem initialized [ 0.825000] ACPI: bus type USB registered [ 0.825000] usbcore: registered new interface driver usbfs [ 0.827118] usbcore: registered new interface driver hub [ 0.830109] usbcore: registered new device driver usb [ 0.832179] pps_core: LinuxPPS API ver. 1 registered [ 0.834026] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.837058] PTP clock support registered [ 0.839346] EDAC MC: Ver: 3.0.0 [ 0.845109] PCI: Using ACPI for IRQ routing [ 0.852247] NetLabel: Initializing [ 0.855019] NetLabel: domain hash size = 128 [ 0.859018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.864108] NetLabel: unlabeled traffic allowed by default [ 0.869655] vgaarb: loaded [ 0.873339] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.874020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.884514] clocksource: Switched to clocksource kvm-clock [ 1.061904] VFS: Disk quotas dquot_6.6.0 [ 1.064256] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.071063] *** VALIDATE ramfs *** [ 1.073722] *** VALIDATE hugetlbfs *** [ 1.077338] pnp: PnP ACPI init [ 1.085534] pnp: PnP ACPI: found 6 devices [ 1.115838] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.119758] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.123174] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.126984] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.131204] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.135550] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.140167] NET: Registered protocol family 2 [ 1.147726] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.161944] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.167234] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.176236] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.182118] TCP: Hash tables configured (established 65536 bind 65536) [ 1.186480] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.191217] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.195051] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.198830] NET: Registered protocol family 1 [ 1.202375] RPC: Registered named UNIX socket transport module. [ 1.206318] RPC: Registered udp transport module. [ 1.208662] RPC: Registered tcp transport module. [ 1.211049] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.215101] NET: Registered protocol family 44 [ 1.217317] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.220532] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.224045] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.227438] PCI: CLS 0 bytes, default 64 [ 1.229910] Unpacking initramfs... [ 3.137595] debug: unmapping init [mem 0xffff92dcfcc54000-0xffff92dcfffbffff] [ 3.146644] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.149239] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.153787] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.803508] Initialise system trusted keyrings [ 3.805357] Key type blacklist registered [ 3.808067] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.819811] zbud: loaded [ 3.823861] *** VALIDATE nfs *** [ 3.826187] *** VALIDATE nfs4 *** [ 3.828463] pstore: using deflate compression [ 3.835251] Platform Keyring initialized [ 3.983243] NET: Registered protocol family 38 [ 3.985921] Key type asymmetric registered [ 3.987933] Asymmetric key parser 'x509' registered [ 3.990277] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.994740] io scheduler mq-deadline registered [ 3.997242] io scheduler kyber registered [ 3.999413] io scheduler bfq registered [ 4.001879] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.006230] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.010171] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.014076] ACPI: Power Button [PWRF] [ 4.022161] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.030750] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.054457] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.065556] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.094101] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.128512] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.161670] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.167996] Non-volatile memory driver v1.3 [ 4.170294] Linux agpgart interface v0.103 [ 4.205934] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 4.211594] vda: detected capacity change from 0 to 76681216 [ 4.243614] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.250664] vdb: detected capacity change from 0 to 1073741824 [ 4.275513] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.280060] vdc: detected capacity change from 0 to 2621440000 [ 4.299313] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.302931] vdd: detected capacity change from 0 to 2621440000 [ 4.326203] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.330994] vde: detected capacity change from 0 to 4294967296 [ 4.352463] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.356265] vdf: detected capacity change from 0 to 4294967296 [ 4.365124] libphy: Fixed MDIO Bus: probed [ 4.373566] usbcore: registered new interface driver usbserial_generic [ 4.378451] usbserial: USB Serial support registered for generic [ 4.384074] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.390090] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.392276] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.395131] mousedev: PS/2 mouse device common for all mice [ 4.399771] rtc_cmos 00:05: RTC can wake from S4 [ 4.404360] rtc_cmos 00:05: registered as rtc0 [ 4.405921] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.407931] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.413338] intel_pstate: CPU model not supported [ 4.422740] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.428001] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.432058] hid: raw HID events driver (C) Jiri Kosina [ 4.436753] usbcore: registered new interface driver usbhid [ 4.440333] usbhid: USB HID core driver [ 4.442747] drop_monitor: Initializing network drop monitor service [ 4.446720] Initializing XFRM netlink socket [ 4.449195] NET: Registered protocol family 10 [ 4.453093] Segment Routing with IPv6 [ 4.454741] NET: Registered protocol family 17 [ 4.457313] mpls_gso: MPLS GSO support [ 4.463460] RAS: Correctable Errors collector initialized. [ 4.466272] AVX version of gcm_enc/dec engaged. [ 4.468872] AES CTR mode by8 optimization enabled [ 4.568500] sched_clock: Marking stable (4568485342, 0)->(5630103512, -1061618170) [ 4.571499] registered taskstats version 1 [ 4.573817] Loading compiled-in X.509 certificates [ 4.576840] zswap: loaded using pool lzo/zbud [ 4.612954] Key type big_key registered [ 4.628484] Key type encrypted registered [ 4.630884] ima: No TPM chip found, activating TPM-bypass! [ 4.633859] ima: Allocated hash algorithm: sha1 [ 4.635910] ima: No architecture policies found [ 4.637939] evm: Initialising EVM extended attributes: [ 4.640198] evm: security.selinux [ 4.641520] evm: security.ima [ 4.642986] evm: security.capability [ 4.644895] evm: HMAC attrs: 0x1 [ 4.648245] rtc_cmos 00:05: setting system clock to 2026-09-03 19:16:02 UTC (1788462962) [ 4.656855] debug: unmapping init [mem 0xffffffffb0a03000-0xffffffffb0bfffff] [ 4.661103] debug: unmapping init [mem 0xffffffffaf782000-0xffffffffafa58fff] [ 4.670125] Write protecting the kernel read-only data: 28672k [ 4.674450] debug: unmapping init [mem 0xffffffffade03000-0xffffffffadffffff] [ 4.678173] debug: unmapping init [mem 0xffffffffae714000-0xffffffffae7fffff] [ 4.717464] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.727379] systemd[1]: Detected virtualization kvm. [ 4.729691] systemd[1]: Detected architecture x86-64. [ 4.732641] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.792351] systemd[1]: No hostname configured. [ 4.795064] systemd[1]: Set hostname to . [ 4.797798] random: systemd: uninitialized urandom read (16 bytes read) [ 4.801352] systemd[1]: Initializing machine ID from random generator. [ 4.900066] random: ln: uninitialized urandom read (6 bytes read) [ 5.208341] random: systemd: uninitialized urandom read (16 bytes read) [ 5.214753] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 5.242864] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ 5.273501] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.830171] device-mapper: uevent: version 1.0.3 [ 7.839715] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ 8.889109] hrtimer: interrupt took 4097528 ns [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 11.153692] virtio_net virtio0 ens2: renamed from eth0 [ 11.356480] random: fast init done [ 11.793697] scsi host0: ata_piix [ 11.903507] scsi host1: ata_piix [ 11.941236] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.951135] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 17.159831] random: crng init done [ 17.161845] random: 7 urandom warning(s) missed due to ratelimiting [ 21.139489] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 23.095804] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 28.635690] printk: systemd: 26 output lines suppressed due to ratelimiting [ 30.561689] SELinux: Disabled at runtime. [ 30.769754] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 30.792397] systemd[1]: Detected virtualization kvm. [ 30.796895] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 33.103531] systemd[1]: initrd-switch-root.service: Succeeded. [ 33.118104] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 33.201205] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 33.232081] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 33.247657] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 33.299047] systemd[1]: Starting Journal Service... Starting Journal Service... [ 33.329613] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [[ 34.219821] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 36.237118] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 37.521644] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 37.697383] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 38.532427] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 39.297485] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit)[ 46.849930] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit)[ 48.040584] NFS: Registering the id_resolver key type [ 48.050664] Key type id_resolver registered [ 48.065347] Key type id_legacy registered [*** ] A start job is running for Configur…only root support (15s / no limit) [** ] A start job is running for Configur…only root support (15s / no limit) [* ] A start job is running for Configur…only root support (16s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg434-server login: [ 110.981841] libcfs: loading out-of-tree module taints kernel. [ 111.027842] Key type ._llcrypt registered [ 111.037274] Key type .llcrypt registered [ 111.151512] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 130.040942] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 131.361769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 131.385452] alg: No test for adler32 (adler32-zlib) [ 132.640208] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 133.786449] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 135.607183] Key type lgssc registered [ 137.575260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 156.867690] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 197.298168] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 209.791383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 209.866944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 211.215733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 211.249789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 211.381339] Lustre: lustre-MDT0000: new disk, initializing [ 211.548948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 211.578904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 215.628255] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.648913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 230.820543] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 230.849546] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 230.855643] Lustre: Skipped 1 previous similar message [ 231.002906] Lustre: lustre-MDT0001: new disk, initializing [ 231.107323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 231.145523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 231.169440] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 237.141179] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 242.828732] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 253.306520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 253.576936] Lustre: lustre-OST0000: new disk, initializing [ 253.581115] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 253.595439] Lustre: 8447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 253.679750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 260.392137] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 262.706919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 262.722803] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 262.854642] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 276.893426] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 277.052682] Lustre: lustre-OST0001: new disk, initializing [ 277.056889] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 277.068502] Lustre: 9522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 277.132145] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 284.929820] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 287.291624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 287.307965] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 287.368476] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 297.587583] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 306.522614] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 313.767550] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing check_logdir /tmp/testlogs/ [ 319.004258] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing yml_node [ 324.420089] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 326.968891] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 330.475427] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 332.332285] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 15:21:28 EDT 2026 [ 350.505844] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 352.347694] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 369.631671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 369.645543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 369.659186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 372.193645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 372.212027] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 372.238821] Lustre: Skipped 2 previous similar messages [ 374.205104] Lustre: server umount lustre-MDT0000 complete [ 377.962214] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788463335 with bad export cookie 14925167741520348619 [ 377.971326] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 377.983255] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 382.432322] LustreError: 6518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 382.432733] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 382.452731] LustreError: 6518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 382.473855] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 382.484620] Lustre: Skipped 4 previous similar messages [ 384.579517] Lustre: server umount lustre-MDT0001 complete [ 389.177837] Lustre: server umount lustre-OST0000 complete [ 393.507585] Lustre: server umount lustre-OST0001 complete [ 401.021585] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 411.308096] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 453.112135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 453.461327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 453.501846] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 453.602897] Lustre: lustre-MDT0000: new disk, initializing [ 453.726359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 453.749730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 458.865273] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 470.122875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 470.253316] Lustre: 15073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 470.343554] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 470.349463] Lustre: Skipped 1 previous similar message [ 470.502274] Lustre: lustre-MDT0001: new disk, initializing [ 470.721449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 470.772855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 470.795163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 476.179188] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 487.961451] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 494.219498] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 501.675646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 502.050718] Lustre: lustre-OST0000: new disk, initializing [ 502.058731] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 502.064594] Lustre: 17044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 502.155436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 503.616150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 503.633311] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 503.790920] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 509.474262] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 519.847751] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 524.831310] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 525.002916] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 530.612549] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 530.919477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 534.497043] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 534.525580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 534.553452] Lustre: Skipped 1 previous similar message [ 534.566489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 534.574542] Lustre: Skipped 1 previous similar message [ 539.620617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 539.629599] Lustre: Skipped 1 previous similar message [ 539.970229] Lustre: server umount lustre-OST0000 complete [ 544.751236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 544.764912] Lustre: Skipped 1 previous similar message [ 549.860025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 549.874147] Lustre: Skipped 3 previous similar messages [ 558.047561] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 558.229603] Lustre: server umount lustre-MDT0000 complete [ 560.096254] LustreError: 15087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 560.107994] LustreError: 15087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 564.195984] LustreError: 15082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 564.225129] LustreError: 15082:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 565.444628] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788463523 with bad export cookie 14925167741520350152 [ 565.455407] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 565.464741] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 566.076311] Lustre: server umount lustre-MDT0001 complete [ 573.055613] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 15:25:28 (1788463528) [ 594.012450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 594.537615] LustreError: 19919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.631969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 599.291598] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 600.033437] LustreError: 19920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 608.459420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 613.483067] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 621.735787] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 628.756994] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 631.403545] Lustre: Failing over lustre-MDT0000 [ 631.851572] Lustre: server umount lustre-MDT0000 complete [ 633.518047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 633.535150] Lustre: Skipped 2 previous similar messages [ 633.539679] LustreError: 19915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 633.558432] LustreError: 19915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 649.761424] Lustre: 3646:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788463591/real 1788463591] req@ffff92dc43771880 x1875339474274176/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788463607 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 649.800847] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 652.754543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 653.924194] LustreError: 19920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 653.959657] LustreError: 19920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 660.516715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 660.523028] Lustre: Skipped 1 previous similar message [ 660.575382] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 665.571436] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 665.576542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 665.604774] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 666.150141] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 676.546421] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 680.928080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 680.943168] LustreError: Skipped 1 previous similar message [ 680.952515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 680.976227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 680.990044] Lustre: Skipped 2 previous similar messages [ 687.065672] Lustre: server umount lustre-MDT0000 complete [ 689.126751] LustreError: 19916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 689.150110] LustreError: 19916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 691.485210] LustreError: 19902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788463649 with bad export cookie 14925167741520351566 [ 691.488120] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 691.508910] LustreError: 19902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 692.015981] Lustre: server umount lustre-MDT0001 complete [ 700.852969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 706.576643] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 715.720425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 721.420308] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 729.554612] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 735.348849] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 737.875792] Lustre: Failing over lustre-MDT0000 [ 738.491426] Lustre: server umount lustre-MDT0000 complete [ 740.962487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 740.980021] Lustre: Skipped 2 previous similar messages [ 756.706672] LustreError: 23260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 756.732839] LustreError: 23260:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 757.731598] Lustre: 3644:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788463698/real 1788463698] req@ffff92dc42c50380 x1875339474304512/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788463714 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 757.785648] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 758.559318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 769.541533] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 774.626468] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 774.642794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 774.672948] Lustre: Skipped 1 previous similar message [ 774.707952] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 775.657379] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 787.519384] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 795.104376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 795.119700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 795.128572] Lustre: Skipped 2 previous similar messages [ 798.717805] Lustre: server umount lustre-MDT0000 complete [ 802.669016] LustreError: 23239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788463760 with bad export cookie 14925167741520352504 [ 802.672724] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 802.677646] LustreError: 23239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 803.193902] Lustre: server umount lustre-MDT0001 complete [ 814.464455] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 15:29:30 (1788463770) [ 836.503996] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 850.200837] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 891.920574] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 903.782592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 904.075483] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 904.127449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 904.249258] Lustre: lustre-MDT0000: new disk, initializing [ 904.324390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 904.333254] Lustre: Skipped 3 previous similar messages [ 904.368614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 909.486963] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 921.213569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 921.340699] Lustre: 29638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 921.671367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 926.678924] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 938.447326] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 945.535628] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 953.465619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 953.911329] Lustre: 31609:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 955.969271] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 956.031316] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 956.103761] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 956.135756] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 961.420212] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 971.885928] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 978.295184] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1002.467699] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1002.468792] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1002.494517] Lustre: Skipped 2 previous similar messages [ 1002.503882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1002.525926] Lustre: Skipped 3 previous similar messages [ 1005.626555] Lustre: server umount lustre-OST0000 complete [ 1015.912988] Lustre: server umount lustre-MDT0000 complete [ 1017.826689] LustreError: 32477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1017.851311] LustreError: 32477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 1019.995572] LustreError: 29630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788463977 with bad export cookie 14925167741520352910 [ 1019.997423] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1020.013212] LustreError: 29630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1020.789754] Lustre: server umount lustre-MDT0001 complete [ 1031.853604] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 1035.237320] Key type lgssc unregistered [ 1035.686598] LNet: 33587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.695390] LNetError: 33587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1035.705814] LNet: Removed LNI 192.168.204.134@tcp [ 1037.004175] Key type .llcrypt unregistered [ 1037.010780] Key type ._llcrypt unregistered [ 1063.436551] Key type ._llcrypt registered [ 1063.442777] Key type .llcrypt registered [ 1063.565802] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 1064.577727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1064.790173] alg: No test for adler32 (adler32-zlib) [ 1065.919472] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 1066.126772] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 1067.815185] Key type lgssc registered [ 1068.994433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1077.176551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1077.210278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1078.788986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1083.619541] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1089.000580] LustreError: 34996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1089.025770] LustreError: 34996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1092.650432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1093.212807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1098.813171] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1107.956460] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1114.376926] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1121.805941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1122.216051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1130.467778] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1137.619808] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1139.527896] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1155.151101] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 15:35:11 (1788464111) [ 1156.973476] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1158.760462] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1165.463082] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 15:35:21 (1788464121) [ 1173.771261] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 15:35:29 (1788464129) [ 1174.899405] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1183.914313] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1183.920912] Lustre: Skipped 2 previous similar messages [ 1185.792726] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 15:35:41 (1788464141) [ 1189.396625] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1189.402380] Lustre: Skipped 3 previous similar messages [ 1206.578114] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1206.588152] Lustre: Skipped 6 previous similar messages [ 1209.333822] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 15:36:05 (1788464165) [ 1212.096703] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1215.807104] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1225.658415] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 15:36:21 (1788464181) [ 1227.613485] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1227.617655] Lustre: Skipped 3 previous similar messages [ 1240.790659] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 15:36:36 (1788464196) [ 1255.394179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1255.399304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1255.416536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1255.426217] Lustre: Skipped 1 previous similar message [ 1255.433676] Lustre: Skipped 1 previous similar message [ 1260.523793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1260.540354] Lustre: Skipped 1 previous similar message [ 1261.559387] Lustre: server umount lustre-OST0000 complete [ 1267.169054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1267.188703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1267.238550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1270.753886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1270.772421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1273.093516] Lustre: server umount lustre-MDT0000 complete [ 1275.872173] LustreError: 35724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1275.906796] LustreError: 35724:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1277.151420] LustreError: 34978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788464235 with bad export cookie 1123836673002768229 [ 1277.154655] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1277.178382] LustreError: 34978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1277.766445] Lustre: server umount lustre-MDT0001 complete [ 1289.632972] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 1293.323759] Key type lgssc unregistered [ 1293.785825] LNet: 42862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1293.799165] LNetError: 42862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1293.819972] LNet: Removed LNI 192.168.204.134@tcp [ 1294.864306] Key type .llcrypt unregistered [ 1294.867728] Key type ._llcrypt unregistered [ 1316.232710] Key type ._llcrypt registered [ 1316.236943] Key type .llcrypt registered [ 1316.351654] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 1333.206911] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 1334.541953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1334.625042] alg: No test for adler32 (adler32-zlib) [ 1335.646257] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 1335.890493] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 1337.553548] Key type lgssc registered [ 1338.988770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1377.885674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1377.917342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1379.219604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1379.282780] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1379.430718] Lustre: lustre-MDT0000: new disk, initializing [ 1379.501180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1379.521093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1384.308558] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1396.281070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1396.368454] Lustre: 46014:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1396.413557] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1396.419084] Lustre: Skipped 1 previous similar message [ 1396.500418] Lustre: lustre-MDT0001: new disk, initializing [ 1396.638163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1396.732452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1396.752958] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1400.814800] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1409.314896] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1414.880911] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1421.641248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1421.878596] Lustre: lustre-OST0000: new disk, initializing [ 1421.884630] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1421.893175] Lustre: 47982:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1422.012478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1428.346737] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1431.125753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1431.132563] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1431.223479] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1438.002372] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1443.952945] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1444.208678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1449.400416] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1449.705350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1452.011409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1452.011489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1452.016652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1452.036752] Lustre: Skipped 1 previous similar message [ 1452.047170] LustreError: Skipped 1 previous similar message [ 1455.018541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1455.027862] Lustre: Skipped 2 previous similar messages [ 1457.934248] Lustre: server umount lustre-OST0000 complete [ 1463.264117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1463.285303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1463.316051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1465.313132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1465.333608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1465.346207] Lustre: Skipped 1 previous similar message [ 1468.009838] Lustre: server umount lustre-MDT0000 complete [ 1471.991645] LustreError: 46005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788464429 with bad export cookie 8076260642381027377 [ 1471.993600] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1472.001995] LustreError: 46005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1472.508535] Lustre: server umount lustre-MDT0001 complete [ 1484.491402] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 15:40:39 (1788464439) [ 1491.909638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1492.408573] LustreError: 50442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1492.430074] LustreError: 50442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1492.502338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1497.123673] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1497.572457] LustreError: 50443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1502.695184] LustreError: 50442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1506.496835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1506.973347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1512.062934] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1519.835983] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1526.594879] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1533.646064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1534.075352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1541.657695] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1550.279338] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1555.100592] LustreError: 52972:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1556.141210] Lustre: Modifying parameter general.timeout=1 in log params [ 1557.935978] Lustre: Modifying parameter general.timeout=3 in log params [ 1557.941612] Lustre: Skipped 1 previous similar message [ 1560.659761] Lustre: Modifying parameter general.timeout=6 in log params [ 1560.668185] Lustre: Skipped 2 previous similar messages [ 1565.379199] Lustre: Modifying parameter general.timeout=11 in log params [ 1565.384554] Lustre: Skipped 4 previous similar messages [ 1573.452225] Lustre: Modifying parameter general.timeout=21 in log params [ 1573.457750] Lustre: Skipped 9 previous similar messages [ 1589.467675] Lustre: Modifying parameter general.timeout=39 in log params [ 1589.474853] Lustre: Skipped 17 previous similar messages [ 1622.810937] Lustre: Modifying parameter general.timeout=70 in log params [ 1622.822549] Lustre: Skipped 30 previous similar messages [ 1687.491570] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 15:44:03 (1788464643) [ 1688.813368] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1688.820699] Lustre: Skipped 60 previous similar messages [ 1690.924027] Lustre: 59912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1692.990460] Lustre: 60011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1695.255627] Lustre: 60126:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1701.256756] Lustre: 60421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1701.280331] Lustre: 60421:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1742.428172] Lustre: 61805:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1742.488579] LustreError: 61805:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1747.676807] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 15:45:03 (1788464703) [ 1767.394929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1767.398401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1767.411933] Lustre: Skipped 2 previous similar messages [ 1767.430801] Lustre: Skipped 2 previous similar messages [ 1769.905144] Lustre: server umount lustre-MDT0000 complete [ 1772.512442] LustreError: 51170:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1772.531422] LustreError: 51170:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1773.453546] LustreError: 50425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788464731 with bad export cookie 8076260642381028259 [ 1773.458045] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1773.472384] LustreError: 50425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1773.997415] Lustre: server umount lustre-MDT0001 complete [ 1788.093410] Lustre: server umount lustre-OST0000 complete [ 1813.357265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1813.735575] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1813.990954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1819.400838] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1828.328339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1828.448426] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1828.471601] Lustre: 64338:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1833.631598] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1841.590686] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1847.015841] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1854.376397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1854.546435] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1854.791625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.800696] Lustre: Skipped 1 previous similar message [ 1862.155823] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1870.043233] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1871.509647] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1871.519829] Lustre: Skipped 22 previous similar messages [ 1871.587663] Lustre: 66906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1879.015050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1879.034063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1879.057182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1884.221601] Lustre: server umount lustre-OST0000 complete [ 1889.766538] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1889.796345] Lustre: Skipped 1 previous similar message [ 1893.885785] Lustre: server umount lustre-MDT0000 complete [ 1894.880164] LustreError: 64345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1894.892679] LustreError: 64345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1897.118484] LustreError: 64330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788464854 with bad export cookie 8076260642381031402 [ 1897.130387] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1897.131486] LustreError: 64330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1897.556338] Lustre: server umount lustre-MDT0001 complete [ 1910.579653] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 1913.840631] Key type lgssc unregistered [ 1914.377247] LNet: 68117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1914.390507] LNetError: 68117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1914.403868] LNet: Removed LNI 192.168.204.134@tcp [ 1915.442473] Key type .llcrypt unregistered [ 1915.450834] Key type ._llcrypt unregistered [ 1942.822620] Key type ._llcrypt registered [ 1942.827834] Key type .llcrypt registered [ 1943.002538] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 1944.296681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1944.342527] alg: No test for adler32 (adler32-zlib) [ 1945.552443] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 1945.975296] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 1947.671190] Key type lgssc registered [ 1948.994791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1958.287690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1958.314363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1960.011473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1965.445064] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1970.145632] LustreError: 69529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1970.173459] LustreError: 69529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1975.265520] LustreError: 69528:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1975.295821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1976.157597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1980.478321] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1987.897126] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1993.956215] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2001.850233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2002.460167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2010.208489] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2018.325512] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2029.688217] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2041.314501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2041.320475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2041.336503] Lustre: Skipped 1 previous similar message [ 2041.356241] Lustre: Skipped 2 previous similar messages [ 2046.169229] Lustre: server umount lustre-MDT0000 complete [ 2046.435233] LustreError: 69528:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2046.454702] LustreError: 69528:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2050.005165] LustreError: 69508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788465007 with bad export cookie 3783421873322139419 [ 2050.011793] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2050.015159] LustreError: 69508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2050.318882] Lustre: server umount lustre-MDT0001 complete [ 2065.840314] Lustre: server umount lustre-OST0000 complete [ 2095.363513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2095.825359] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2095.915948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2096.221835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2101.504722] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2111.364348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2111.437083] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2111.477329] Lustre: 74404:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2112.089520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2118.389883] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2127.408729] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2134.082783] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2142.665458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2143.091382] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2143.766375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2151.921415] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2160.875139] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2163.320144] Lustre: 76980:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2166.020581] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2175.456962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2175.468508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2175.470350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2175.481313] Lustre: Skipped 2 previous similar messages [ 2175.508194] Lustre: Skipped 1 previous similar message [ 2177.943586] Lustre: server umount lustre-OST0000 complete [ 2183.650516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2183.659642] LustreError: Skipped 1 previous similar message [ 2183.664535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2183.680545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2185.699076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2185.702585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2189.113162] Lustre: server umount lustre-MDT0000 complete [ 2190.818837] LustreError: 74412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2190.845782] LustreError: 74412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2194.543805] LustreError: 74396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788465152 with bad export cookie 3783421873322140266 [ 2194.548214] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2194.556907] LustreError: 74396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2195.066184] Lustre: server umount lustre-MDT0001 complete [ 2208.211316] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 2211.085347] Key type lgssc unregistered [ 2211.511383] LNet: 78188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2211.514891] LNetError: 78188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2212.586387] LNet: Removed LNI 192.168.204.134@tcp [ 2213.522384] Key type .llcrypt unregistered [ 2213.524928] Key type ._llcrypt unregistered [ 2245.518910] Key type ._llcrypt registered [ 2245.521593] Key type .llcrypt registered [ 2245.638340] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 2246.649731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2246.687343] alg: No test for adler32 (adler32-zlib) [ 2248.014408] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 2248.350535] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 2250.015251] Key type lgssc registered [ 2251.490361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2261.064368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2261.088920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2262.851514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2269.497665] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2272.232166] LustreError: 79597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2272.263213] LustreError: 79597:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2277.349736] LustreError: 79598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2281.328733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2282.332857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2289.130498] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2297.350916] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2303.789633] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2310.513262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2311.276718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2318.505576] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2327.191854] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2338.531651] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2339.469722] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2341.402115] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2341.409485] Lustre: Skipped 1 previous similar message [ 2344.537768] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2344.545161] Lustre: Skipped 2 previous similar messages [ 2348.662942] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2348.670942] Lustre: Skipped 3 previous similar messages [ 2356.714661] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2356.717443] Lustre: Skipped 7 previous similar messages [ 2372.890890] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2372.900397] Lustre: Skipped 15 previous similar messages [ 2405.481844] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2405.488298] Lustre: Skipped 31 previous similar messages [ 2446.681562] Lustre: 87705:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2449.696835] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 15:56:45 (1788465405) [ 2451.650924] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2453.575602] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 15:56:49 (1788465409) [ 2460.130094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2460.136730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2460.157147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2461.152116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2461.157013] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2461.157567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2464.370419] Lustre: server umount lustre-OST0000 complete [ 2471.394891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2471.398905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.410973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2471.410984] Lustre: Skipped 1 previous similar message [ 2471.430261] Lustre: Skipped 1 previous similar message [ 2474.939142] Lustre: server umount lustre-MDT0000 complete [ 2476.516051] LustreError: 79598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2476.535945] LustreError: 79598:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2479.409758] LustreError: 79578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788465437 with bad export cookie 10751219926760151462 [ 2479.412216] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2479.419372] LustreError: 79578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2479.983177] Lustre: server umount lustre-MDT0001 complete [ 2492.612642] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 2496.303706] Key type lgssc unregistered [ 2496.730834] LNet: 88953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2496.753321] LNetError: 88953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2496.789836] LNet: Removed LNI 192.168.204.134@tcp [ 2497.949486] Key type .llcrypt unregistered [ 2497.953694] Key type ._llcrypt unregistered [ 2504.688727] Key type ._llcrypt registered [ 2504.691643] Key type .llcrypt registered [ 2504.869239] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2510.826910] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules [ 2511.997494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2512.044104] alg: No test for adler32 (adler32-zlib) [ 2513.218078] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 2513.832154] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 2515.751176] Key type lgssc registered [ 2517.799626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2519.702392] LustreError: 90176:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2539.160260] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 2540.959966] LustreError: 90176:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2547.808813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2547.822864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2549.278917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2553.940425] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2559.457541] LustreError: 91166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.489990] LustreError: 91166:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2563.713131] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 15:58:39 (1788465519) [ 2564.577163] LustreError: 91167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2569.276066] LustreError: 91198:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2569.289744] LustreError: 91198:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2569.307951] LustreError: 91198:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2569.820731] Lustre: server umount lustre-MDT0000 complete [ 2584.171351] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 2587.388815] Key type lgssc unregistered [ 2587.753855] LNet: 92691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2587.762485] LNetError: 92691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2587.789765] LNet: Removed LNI 192.168.204.134@tcp [ 2588.652440] Key type .llcrypt unregistered [ 2588.663654] Key type ._llcrypt unregistered [ 2606.151707] Key type ._llcrypt registered [ 2606.154655] Key type .llcrypt registered [ 2606.282443] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 2607.240690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2607.388027] alg: No test for adler32 (adler32-zlib) [ 2608.481044] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 2608.745384] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 2610.431194] Key type lgssc registered [ 2611.384862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2619.008719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2619.020230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2620.572128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2624.721279] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2631.150262] LustreError: 93905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2631.187954] LustreError: 93905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2633.842162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2634.466070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2639.484186] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2647.534949] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2653.477806] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2660.735242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2661.538637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2668.117086] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2674.777132] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2721.176200] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 16:01:17 (1788465677) [ 2722.678415] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2724.428617] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 16:01:20 (1788465680) [ 2730.469259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2730.470940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2730.485063] Lustre: Skipped 1 previous similar message [ 2730.498066] Lustre: Skipped 2 previous similar messages [ 2734.383297] Lustre: server umount lustre-MDT0000 complete [ 2735.585285] LustreError: 93904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2735.600719] LustreError: 93904:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2737.935360] LustreError: 93886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788465695 with bad export cookie 14505255723466905740 [ 2737.940343] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2737.956963] LustreError: 93886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2738.175513] Lustre: server umount lustre-MDT0001 complete [ 2752.297965] Lustre: server umount lustre-OST0000 complete [ 2762.117649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2762.503612] LustreError: 98195:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2762.650196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2766.711805] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2767.850830] LustreError: 98196:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2772.974041] LustreError: 98195:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2774.434571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2774.895668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2778.281498] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2784.073893] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2788.579870] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2798.427415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2798.641748] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2798.648395] LustreError: 98184:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2798.670360] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2798.676880] LustreError: 100278:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2798.682959] LustreError: 100278:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2798.687892] LustreError: 100278:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2798.747254] Lustre: server umount lustre-OST0000 complete [ 2798.752417] LustreError: 100278:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2806.064938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2806.295633] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2806.305941] LustreError: 98184:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2806.335304] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2806.344406] LustreError: 100743:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2806.352586] LustreError: 100743:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2806.364569] LustreError: 100743:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2806.449352] Lustre: server umount lustre-OST0000 complete [ 2806.455805] LustreError: 100743:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2815.315052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2815.580306] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2815.588039] Lustre: Found index 0 for lustre-OST0000, updating log [ 2815.592694] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2815.677304] Lustre: lustre-OST0000: new disk, initializing [ 2815.682236] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2815.692265] Lustre: 101277:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2816.137372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2823.341462] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2826.820176] Lustre: Failing over lustre-OST0000 [ 2826.995499] Lustre: server umount lustre-OST0000 complete [ 2830.817982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2830.834574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2830.858130] Lustre: Skipped 1 previous similar message [ 2830.866128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2834.913272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2834.918137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2837.101483] Lustre: server umount lustre-MDT0000 complete [ 2840.054427] LustreError: 98195:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2840.078760] LustreError: 98195:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2841.211558] LustreError: 98177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788465799 with bad export cookie 14505255723466908484 [ 2841.212854] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2841.241068] LustreError: 98177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2852.032091] Lustre: server umount lustre-MDT0001 complete [ 2863.117937] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 16:03:38 (1788465818) [ 2887.079175] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 2898.401436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2898.935211] LustreError: 103626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2899.050934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2903.463390] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2912.095356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2917.120332] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2920.455805] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2926.805084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2927.444649] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2933.627394] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2937.703227] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2937.741861] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2937.747856] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2943.968069] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2944.107814] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2944.193397] Lustre: lustre-OST0001: new disk, initializing [ 2944.199102] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2944.205934] Lustre: 105945:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2944.522914] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2944.526617] Lustre: Skipped 2 previous similar messages [ 2950.302126] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2953.797047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2953.821550] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2953.953743] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2957.734918] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2964.251569] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2969.057403] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2969.066258] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2969.087317] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2969.092331] Lustre: Skipped 1 previous similar message [ 2973.255715] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.34@tcp (stopping) [ 2973.269703] Lustre: Skipped 2 previous similar messages [ 2983.391201] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2983.457203] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.34@tcp (stopping) [ 2983.474375] Lustre: Skipped 7 previous similar messages [ 2983.596439] Lustre: server umount lustre-MDT0001 complete [ 2984.932466] LustreError: 103626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2984.951194] LustreError: 103626:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2993.837410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2993.985981] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2993.989830] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2994.033079] Lustre: 103616:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2995.841617] LustreError: 107495:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2995.846890] LustreError: 107495:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2995.870067] LustreError: 103619:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2995.958069] LustreError: 107504:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.134@tcp: cfg command failed: rc = -17 [ 2995.976654] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2995.976654] [ 3000.340701] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3000.694123] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3016.118356] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 16:06:11 (1788465971) [ 3039.210461] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 3064.323904] Lustre: 109567:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3092.404839] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3095.891236] Lustre: 110702:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3184.096539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3184.118278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3184.141751] Lustre: Skipped 2 previous similar messages [ 3184.156459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3187.356773] Lustre: server umount lustre-MDT0000 complete [ 3189.731230] LustreError: 110998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3189.761148] LustreError: 110998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3191.795720] LustreError: 103609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788466149 with bad export cookie 14505255723466909485 [ 3191.802225] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3191.822217] LustreError: 103609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3192.119832] Lustre: server umount lustre-MDT0001 complete [ 3206.778975] Lustre: server umount lustre-OST0000 complete [ 3210.207592] Lustre: 93215:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788466152/real 1788466152] req@ffff92dc43a43100 x1875342069692288/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788466168 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3210.280932] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3210.297977] Lustre: Skipped 3 previous similar messages [ 3210.845732] Lustre: server umount lustre-OST0001 complete [ 3216.224827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3226.658160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3235.380839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3245.199252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3271.356665] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 3283.207254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3283.249975] Lustre: lustre-MDT0000: reset Object Index mappings [ 3284.199171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3284.216324] Lustre: Skipped 1 previous similar message [ 3289.328256] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3300.229282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3300.252225] Lustre: lustre-MDT0001: reset Object Index mappings [ 3307.924047] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3312.005628] Lustre: 115808:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3320.354569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3325.618741] LustreError: 116162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3325.649624] LustreError: 116162:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3325.657504] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 3328.715960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3330.028345] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3339.452666] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3345.414696] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3345.416547] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3346.732336] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3354.405825] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3358.023423] Lustre: 117684:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3405.427910] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 16:12:41 (1788466361) [ 3433.956666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3433.970685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3433.999535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3434.004236] Lustre: Skipped 3 previous similar messages [ 3441.016167] Lustre: server umount lustre-MDT0000 complete [ 3445.113461] LustreError: 114638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788466402 with bad export cookie 14505255723466956336 [ 3445.127521] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3445.137219] LustreError: 114638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3451.681666] Lustre: server umount lustre-MDT0001 complete [ 3461.918608] Lustre: server umount lustre-OST0000 complete [ 3472.335110] Lustre: server umount lustre-OST0001 complete [ 3479.615738] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 3489.958281] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 3528.037756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3528.228826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3528.250781] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3528.313972] Lustre: lustre-MDT0000: new disk, initializing [ 3528.424179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3528.434401] Lustre: Skipped 3 previous similar messages [ 3528.449639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3533.196272] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3539.842962] Lustre: server umount lustre-MDT0000 complete [ 3549.547649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3549.989082] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3550.007436] Lustre: 121949:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3550.024212] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3555.616932] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3561.480849] Lustre: server umount lustre-MDT0000 complete [ 3570.948966] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 16:15:27 (1788466527) [ 3572.643897] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3574.546936] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 16:15:30 (1788466530) [ 3591.633763] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 3601.050047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3606.791944] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3614.897423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3615.043702] Lustre: 123998:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3615.089688] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3615.094902] Lustre: Skipped 1 previous similar message [ 3615.171681] Lustre: lustre-MDT0001: new disk, initializing [ 3615.318124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3615.336229] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3620.041977] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3624.406516] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3630.491340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3630.738212] Lustre: lustre-OST0000: new disk, initializing [ 3630.752038] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3630.763436] Lustre: 125575:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3632.769118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3632.787755] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3632.910391] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3637.308984] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3647.808785] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3647.943403] Lustre: lustre-OST0001: new disk, initializing [ 3647.946594] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3647.951852] Lustre: 126447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3649.315976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3649.328431] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3649.403488] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3654.197389] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3662.695484] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3667.184418] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4058.255960] Lustre: 132862:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4058.263678] Lustre: 132862:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4058.269743] Lustre: 132862:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4058.281803] Lustre: 132862:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4058.289371] Lustre: 132862:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4058.295115] Lustre: 132862:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4307.089874] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:27:42 (1788467262) [ 4325.354012] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4325.359633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4325.387249] Lustre: Skipped 8 previous similar messages [ 4325.397438] Lustre: Skipped 11 previous similar messages [ 4328.779825] Lustre: server umount lustre-MDT0000 complete [ 4330.470117] LustreError: 127232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4330.489855] LustreError: 127232:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 4331.718647] LustreError: 132875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788467289 with bad export cookie 14505255723467024061 [ 4331.722876] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4331.736341] LustreError: 132875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4335.591118] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4335.611287] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4335.615824] Lustre: Skipped 1 previous similar message [ 4346.338798] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4346.662054] Lustre: server umount lustre-MDT0001 complete [ 4356.723635] Lustre: server umount lustre-OST0000 complete [ 4366.013226] Lustre: server umount lustre-OST0001 complete [ 4371.754279] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 4379.759590] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 4408.352748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4408.506991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4408.526648] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4408.599147] Lustre: lustre-MDT0000: new disk, initializing [ 4408.685521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4408.691455] Lustre: Skipped 5 previous similar messages [ 4408.704547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4411.983695] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 4420.398665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4420.486294] Lustre: 135905:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4420.542263] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4420.545598] Lustre: Skipped 1 previous similar message [ 4420.612641] Lustre: lustre-MDT0001: new disk, initializing [ 4420.787392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4420.810394] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4424.346649] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 4431.504596] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4435.756935] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4440.802104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4441.002263] Lustre: lustre-OST0000: new disk, initializing [ 4441.005155] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4441.015696] Lustre: 137870:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4441.106857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4441.113052] Lustre: Skipped 1 previous similar message [ 4442.917221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4442.934987] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4443.026450] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4446.411905] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 4454.821281] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4461.982120] Lustre: lustre-MDD0000: changelog on [ 4464.044282] Lustre: lustre-MDD0001: changelog on [ 6214.648721] Lustre: DEBUG MARKER: conf-sanity test_135: @@@@@@ FAIL: creation thread is running. Is changelog reader stuck? [ 6232.363986] Lustre: lustre-MDD0001: changelog off [ 6235.635851] Lustre: lustre-MDD0000: changelog off [ 6250.466705] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6250.485589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6250.492879] LustreError: Skipped 1 previous similar message [ 6250.493893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6250.493899] Lustre: Skipped 4 previous similar messages [ 6250.567077] Lustre: Skipped 2 previous similar messages [ 6253.897619] Lustre: server umount lustre-OST0000 complete [ 6263.777726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6263.793285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6263.823710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6263.835667] Lustre: Skipped 1 previous similar message [ 6265.833883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6267.689325] Lustre: server umount lustre-MDT0000 complete [ 6270.943686] LustreError: 139323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6270.977694] LustreError: 139323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 6272.427732] LustreError: 136698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788469230 with bad export cookie 14505255723467799724 [ 6272.430100] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6272.444816] LustreError: 136698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6272.671532] Lustre: server umount lustre-MDT0001 complete [ 6285.655855] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 6288.995606] Key type lgssc unregistered [ 6289.553655] LNet: 146756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6289.569532] LNetError: 146756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6289.631485] LNet: Removed LNI 192.168.204.134@tcp [ 6290.834172] Key type .llcrypt unregistered [ 6290.837072] Key type ._llcrypt unregistered [ 6313.786218] Key type ._llcrypt registered [ 6313.787736] Key type .llcrypt registered [ 6313.912338] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 6334.569373] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 6336.984945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6337.144813] alg: No test for adler32 (adler32-zlib) [ 6338.553510] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 6338.879717] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 6340.567688] Key type lgssc registered [ 6342.474311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6387.034387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6387.080090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6388.474417] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6388.549734] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6388.625221] Lustre: lustre-MDT0000: new disk, initializing [ 6388.717709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6388.753218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6395.479338] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6409.091653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6409.296953] Lustre: 149969:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6409.416540] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6409.426458] Lustre: Skipped 1 previous similar message [ 6409.623095] Lustre: lustre-MDT0001: new disk, initializing [ 6409.795601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6409.882670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6409.928430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6415.933335] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6425.612914] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6431.327203] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6438.175626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6438.506879] Lustre: lustre-OST0000: new disk, initializing [ 6438.520279] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6438.536836] Lustre: 151939:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6438.643383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6445.161649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6445.176549] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6445.295850] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6445.867406] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6457.206341] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6464.751410] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_module obdecho/obdecho [ 6467.533885] LustreError: 152943:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6467.566631] LustreError: 152943:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6469.495057] LustreError: 153039:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6486.501100] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6486.506816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6486.524573] Lustre: Skipped 1 previous similar message [ 6486.525629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6486.534975] Lustre: Skipped 1 previous similar message [ 6491.199482] Lustre: server umount lustre-MDT0000 complete [ 6494.986460] LustreError: 151938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788469452 with bad export cookie 10302279590567200411 [ 6494.996823] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6495.005766] LustreError: 151938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6495.545796] Lustre: server umount lustre-MDT0001 complete [ 6509.430879] Lustre: server umount lustre-OST0000 complete [ 6534.165308] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 6545.293984] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 6591.092811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6591.486954] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6591.581274] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6591.686491] Lustre: lustre-MDT0000: new disk, initializing [ 6591.825165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6591.847995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6597.778602] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6613.566615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6613.799115] Lustre: 156758:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6613.871224] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6613.877566] Lustre: Skipped 1 previous similar message [ 6614.096240] Lustre: lustre-MDT0001: new disk, initializing [ 6614.378131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6614.491583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6614.498750] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6620.940718] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6634.355735] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6641.992270] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6650.575604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6650.868241] Lustre: lustre-OST0000: new disk, initializing [ 6650.885787] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6650.903386] Lustre: 158730:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6651.017539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6658.629684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6658.647863] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6658.775911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6659.336611] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6669.775913] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6676.374786] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6676.748467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6684.630691] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6684.990581] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6689.762202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6689.781643] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6689.815164] Lustre: Skipped 1 previous similar message [ 6689.830924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6689.882566] Lustre: Skipped 1 previous similar message [ 6694.486652] Lustre: server umount lustre-OST0000 complete [ 6701.537290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6701.546437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6701.586263] Lustre: Skipped 1 previous similar message [ 6701.598729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6705.120532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6705.125031] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6706.825379] Lustre: server umount lustre-MDT0000 complete [ 6709.280802] LustreError: 156769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6709.302410] LustreError: 156769:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6712.021136] LustreError: 156748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788469669 with bad export cookie 10302279590567201370 [ 6712.022563] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6712.040994] LustreError: 156748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6712.478898] Lustre: server umount lustre-MDT0001 complete [ 6717.043242] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 17:07:52 (1788469672) [ 6719.050135] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6720.954909] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 17:07:56 (1788469676) [ 6729.337586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6730.049297] LustreError: 161145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6730.166352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6735.029584] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6735.332262] LustreError: 161146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6739.428944] LustreError: 161145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6744.548128] LustreError: 161146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6745.106954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6751.392791] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6761.897764] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6768.640570] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6777.259986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6777.737670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6777.748227] Lustre: Skipped 1 previous similar message [ 6785.918653] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6794.606888] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6809.568820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6809.575213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6809.597909] Lustre: Skipped 2 previous similar messages [ 6809.618174] Lustre: Skipped 3 previous similar messages [ 6812.504541] Lustre: server umount lustre-MDT0000 complete [ 6814.691824] LustreError: 161142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6814.711783] LustreError: 161142:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6816.936649] LustreError: 161127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788469774 with bad export cookie 10302279590567202252 [ 6816.951358] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6816.964173] LustreError: 161127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6819.810378] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6819.842996] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6819.863357] Lustre: Skipped 1 previous similar message [ 6823.638523] Lustre: server umount lustre-MDT0001 complete [ 6832.364847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6848.067195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6848.467495] LustreError: 165010:0:(llog.c:1655:llog_backup()) MGC192.168.204.134@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6848.476964] Lustre: 165010:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.134@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6850.529338] LustreError: 165015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6850.560712] LustreError: 165015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6887.395032] LustreError: 165017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6887.415723] LustreError: 165017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [ 6909.408217] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6909.414493] Lustre: 165010:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.134@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6950.435589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6951.999109] LustreError: 165015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6952.052543] LustreError: 165015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 43 previous similar messages [ 6955.496542] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6957.883348] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6968.245331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6978.031082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6992.916211] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6994.236312] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7004.733220] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7006.844819] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7013.609379] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7016.360544] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7038.969507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7038.972960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7039.012428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7039.038829] Lustre: Skipped 2 previous similar messages [ 7039.106812] Lustre: Skipped 4 previous similar messages [ 7044.829853] Lustre: server umount lustre-MDT0000 complete [ 7052.270144] LustreError: 161126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470010 with bad export cookie 10302279590567208741 [ 7052.276144] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7052.299516] LustreError: 161126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7053.029127] Lustre: server umount lustre-MDT0001 complete [ 7071.657567] Lustre: server umount lustre-OST0000 complete [ 7100.744333] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 7112.769464] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 7159.282682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7159.580616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7159.613565] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7159.693142] Lustre: lustre-MDT0000: new disk, initializing [ 7159.764216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7159.767450] Lustre: Skipped 1 previous similar message [ 7159.781831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7165.027760] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7179.300213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7179.514540] Lustre: 170123:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7180.012234] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7186.816055] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7198.505586] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7205.126627] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7212.407708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7212.765609] Lustre: lustre-OST0000: new disk, initializing [ 7212.772725] Lustre: Skipped 1 previous similar message [ 7212.786485] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7212.793956] Lustre: Skipped 2 previous similar messages [ 7212.809603] Lustre: 172093:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7219.723753] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7219.793753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7219.805237] Lustre: Skipped 1 previous similar message [ 7219.812390] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7219.948657] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7231.118418] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7238.162894] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7238.582515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7246.734204] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7247.078615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7250.913470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7250.919628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7250.927207] LustreError: Skipped 1 previous similar message [ 7250.927884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7250.927887] Lustre: Skipped 3 previous similar messages [ 7250.983158] Lustre: Skipped 1 previous similar message [ 7255.983677] Lustre: server umount lustre-OST0000 complete [ 7267.765137] Lustre: server umount lustre-MDT0000 complete [ 7270.397656] LustreError: 170135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7270.430852] LustreError: 170135:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 35 previous similar messages [ 7272.627839] LustreError: 170117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470230 with bad export cookie 10302279590567225541 [ 7272.631294] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7272.637741] LustreError: 170117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7273.244892] Lustre: server umount lustre-MDT0001 complete [ 7286.951509] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 17:17:22 (1788470242) [ 7296.731121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7303.315248] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7316.810101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7325.531815] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7335.201811] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7343.829923] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7352.269767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7361.964692] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7373.206646] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7388.314915] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 17:19:04 (1788470344) [ 7398.909734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7398.912395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7398.924091] LustreError: Skipped 2 previous similar messages [ 7398.925810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7398.944695] Lustre: Skipped 3 previous similar messages [ 7398.970839] Lustre: Skipped 5 previous similar messages [ 7400.229221] Lustre: server umount lustre-OST0000 complete [ 7419.359352] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7419.886627] Lustre: server umount lustre-MDT0000 complete [ 7425.442221] LustreError: 174489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470383 with bad export cookie 10302279590567226423 [ 7425.449256] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7425.479487] LustreError: 174489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7426.309614] Lustre: server umount lustre-MDT0001 complete [ 7440.690996] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 7445.321180] Key type lgssc unregistered [ 7445.787164] LNet: 178244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7445.797473] LNetError: 178244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7445.832908] LNet: Removed LNI 192.168.204.134@tcp [ 7447.080070] Key type .llcrypt unregistered [ 7447.082114] Key type ._llcrypt unregistered [ 7478.174411] Key type ._llcrypt registered [ 7478.178379] Key type .llcrypt registered [ 7478.314737] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 7480.837833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7480.976366] alg: No test for adler32 (adler32-zlib) [ 7482.472118] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 7483.000559] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 7484.815185] Key type lgssc registered [ 7486.610000] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7496.584823] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7496.650943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7512.351613] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7517.471401] LustreError: 179491:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.134@tcp: failed processing log, type 4: rc = -110 [ 7522.591508] LustreError: 179491:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7522.607564] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7522.622756] LustreError: MGC192.168.204.134@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7522.642572] LustreError: 179491:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7522.666238] LustreError: 179491:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 7522.677597] LustreError: 179491:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7522.761327] Lustre: server umount lustre-OST0000 complete [ 7522.766894] LustreError: 179491:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7534.242092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7535.726139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7541.530802] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7545.313119] LustreError: 179991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7545.340582] LustreError: 179991:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7550.434146] LustreError: 179992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7550.892796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7551.260895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7556.596158] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7564.289574] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7570.569934] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7578.549505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7578.776245] LustreError: 181806:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7578.796168] Lustre: 181806:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.134@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7579.052873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7586.203707] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7595.763537] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7601.634715] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7601.650333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7601.694035] Lustre: Skipped 1 previous similar message [ 7601.727428] Lustre: Skipped 2 previous similar messages [ 7606.760949] LustreError: 179992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7606.813267] LustreError: 179992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7606.837947] Lustre: server umount lustre-MDT0000 complete [ 7611.874691] LustreError: 179991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7613.099459] LustreError: 179971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470570 with bad export cookie 17473176759670346493 [ 7613.113155] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7613.129526] LustreError: 179971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7614.048344] LustreError: 180721:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000010d795c8 x1875347180125696/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7614.767820] Lustre: server umount lustre-MDT0001 complete [ 7630.341583] Lustre: server umount lustre-OST0000 complete [ 7642.461542] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 7654.681588] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 7699.160144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7699.535917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7699.590864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7699.708470] Lustre: lustre-MDT0000: new disk, initializing [ 7699.867583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7699.904364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7705.237067] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7717.492324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7717.676042] Lustre: 185336:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7717.776918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7717.787270] Lustre: Skipped 1 previous similar message [ 7717.960661] Lustre: lustre-MDT0001: new disk, initializing [ 7718.102597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7718.140027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7718.157397] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7723.652429] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7734.159945] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7742.343566] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7751.237684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7751.662911] Lustre: lustre-OST0000: new disk, initializing [ 7751.668769] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7751.675974] Lustre: 187305:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7751.789341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7756.901484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7756.922836] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7757.085554] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7760.415854] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7772.972941] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7780.817493] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7781.271600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7788.310867] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7788.547859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7793.134185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7793.147371] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7793.178174] Lustre: Skipped 1 previous similar message [ 7793.183126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7797.075312] Lustre: server umount lustre-OST0000 complete [ 7803.367979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7803.378344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7803.402149] Lustre: Skipped 2 previous similar messages [ 7803.411032] Lustre: Skipped 2 previous similar messages [ 7808.481346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7808.504172] Lustre: Skipped 1 previous similar message [ 7813.600253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7813.609840] Lustre: Skipped 1 previous similar message [ 7817.183227] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7817.550814] Lustre: server umount lustre-MDT0000 complete [ 7818.735466] LustreError: 185341:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7818.761978] LustreError: 185341:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7822.489264] LustreError: 185328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470780 with bad export cookie 17473176759670347214 [ 7822.494543] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7822.505913] LustreError: 185328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7823.003600] Lustre: server umount lustre-MDT0001 complete [ 7833.607855] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 17:26:29 (1788470789) [ 7856.069987] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 7859.688593] Key type lgssc unregistered [ 7860.287469] LNet: 190255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7860.297964] LNetError: 190255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7860.342348] LNet: Removed LNI 192.168.204.134@tcp [ 7861.686705] Key type .llcrypt unregistered [ 7861.688588] Key type ._llcrypt unregistered [ 7885.218135] Key type ._llcrypt registered [ 7885.223401] Key type .llcrypt registered [ 7885.429302] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 7887.060274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7887.125847] alg: No test for adler32 (adler32-zlib) [ 7888.209845] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 7888.510837] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 7890.255405] Key type lgssc registered [ 7891.651362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7901.155073] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7901.203676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7902.886328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7908.629723] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7912.418653] LustreError: 191470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7912.438409] LustreError: 191470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7917.550991] LustreError: 191471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7918.615312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7919.169123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7925.567678] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7934.138897] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7940.814267] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7951.435606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7951.796563] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7951.799629] LustreError: 193336:0:(llog.c:1696:llog_backup()) MGC192.168.204.134@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7951.807234] Lustre: 193336:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.134@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7951.990991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7960.225882] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7970.351505] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7975.398636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7975.419601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7975.441473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7977.955322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7977.971574] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7978.005749] Lustre: Skipped 1 previous similar message [ 7983.074179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7983.084964] Lustre: Skipped 4 previous similar messages [ 7987.170102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7987.395773] Lustre: server umount lustre-MDT0000 complete [ 7988.194627] LustreError: 191467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7988.216682] LustreError: 191467:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7992.012280] LustreError: 191451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470949 with bad export cookie 8575769390248010386 [ 7992.020493] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7992.030088] LustreError: 191451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7992.654632] Lustre: server umount lustre-MDT0001 complete [ 8007.483125] Lustre: server umount lustre-OST0000 complete [ 8019.513356] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 8032.254626] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 8080.132126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8080.571731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8080.816315] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8080.998567] Lustre: lustre-MDT0000: new disk, initializing [ 8081.165481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8081.223860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8087.934248] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8101.490634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8101.605316] Lustre: 196869:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8101.680084] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8101.682577] Lustre: Skipped 1 previous similar message [ 8101.685638] LustreError: 197653:0:(llog.c:1696:llog_backup()) MGC192.168.204.134@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8101.690983] LustreError: 197653:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8101.695829] Lustre: 197653:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.134@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8101.715071] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8101.718222] Lustre: Skipped 1 previous similar message [ 8101.771774] Lustre: lustre-MDT0001: new disk, initializing [ 8101.893161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8101.921782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8101.929774] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8107.042623] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8118.030708] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8124.710630] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8131.811927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8132.127427] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8132.130611] Lustre: Skipped 1 previous similar message [ 8132.133643] LustreError: 198814:0:(llog.c:1696:llog_backup()) MGC192.168.204.134@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8132.140770] LustreError: 198814:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8132.145559] Lustre: 198814:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.134@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8132.161157] Lustre: 198814:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8132.184495] Lustre: lustre-OST0000: new disk, initializing [ 8132.190594] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8132.200533] Lustre: 198840:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8132.315871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8138.867464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8138.890412] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8139.144686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8140.794311] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8152.466705] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8159.096993] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8159.514147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8166.848493] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8167.225655] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8175.083508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8175.084346] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8175.095500] LustreError: Skipped 1 previous similar message [ 8175.102908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8175.118622] Lustre: Skipped 1 previous similar message [ 8176.761573] Lustre: server umount lustre-OST0000 complete [ 8183.784760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8183.797073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8183.825697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8183.842039] Lustre: Skipped 1 previous similar message [ 8195.560328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8195.569083] Lustre: Skipped 5 previous similar messages [ 8196.063318] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8196.301283] Lustre: server umount lustre-MDT0000 complete [ 8200.629038] LustreError: 196861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471158 with bad export cookie 8575769390248011107 [ 8200.648248] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8201.164575] Lustre: server umount lustre-MDT0001 complete [ 8211.674649] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 17:32:47 (1788471167) [ 8232.171887] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 8245.223823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8246.159441] LustreError: 201972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8246.278217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8251.359925] LustreError: 201973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8252.114802] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8261.603075] LustreError: 201972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8261.630300] LustreError: 201972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8263.749672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8271.375501] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8275.919524] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8284.107801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8284.403169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8284.410228] Lustre: Skipped 1 previous similar message [ 8293.165436] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8303.268308] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8303.437352] Lustre: lustre-OST0001: new disk, initializing [ 8303.441263] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8303.448592] Lustre: 204280:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8310.430329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8310.446312] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8310.595798] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8312.291298] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8324.524355] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8335.496923] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8337.314747] Lustre: DEBUG MARKER: ADD OST3 [ 8342.052597] loop: module loaded [ 8342.186614] loop0: detected capacity change from 0 to 204800000 [ 8342.428916] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8342.446928] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8342.458129] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8348.284296] loop0: detected capacity change from 0 to 204800000 [ 8349.223107] Lustre: DEBUG MARKER: STOP OST3 [ 8354.519722] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8354.746335] Lustre: lustre-OST0003: new disk, initializing [ 8354.750154] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8354.756330] Lustre: 206281:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8354.870769] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8354.877114] Lustre: Skipped 1 previous similar message [ 8356.180323] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8356.366766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8356.375111] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8356.449925] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8358.239369] LustreError: 206388:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8360.104771] Lustre: Failing over lustre-OST0003 [ 8360.304471] Lustre: server umount lustre-OST0003 complete [ 8361.451548] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8361.469826] Lustre: Skipped 1 previous similar message [ 8361.481329] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8361.492411] LustreError: 203461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8363.074628] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8372.740816] Lustre: DEBUG MARKER: START OST3 again [ 8380.251916] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8380.650814] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8381.968609] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8382.315029] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8382.318749] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8388.635674] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8411.155961] Lustre: Failing over lustre-OST0003 [ 8411.444521] Lustre: server umount lustre-OST0003 complete [ 8411.642921] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8411.662601] Lustre: Skipped 1 previous similar message [ 8411.673119] LustreError: 203461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8411.701096] LustreError: 203461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 8432.111715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8432.114195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8432.143753] Lustre: Skipped 3 previous similar messages [ 8432.161522] Lustre: Skipped 3 previous similar messages [ 8436.775934] Lustre: server umount lustre-MDT0000 complete [ 8442.181598] LustreError: 201952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471400 with bad export cookie 8575769390248011996 [ 8442.189090] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8442.200423] LustreError: 201952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8448.799608] Lustre: server umount lustre-MDT0001 complete [ 8453.083269] Lustre: server umount lustre-OST0000 complete [ 8457.842139] Lustre: server umount lustre-OST0001 complete [ 8465.872979] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 8475.853509] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 8521.047773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8521.423146] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8521.441553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8521.509333] Lustre: lustre-MDT0000: new disk, initializing [ 8521.585745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8521.594629] Lustre: Skipped 1 previous similar message [ 8521.607774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8527.303726] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8541.280631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8541.551677] Lustre: 211746:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8541.649838] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8541.661726] Lustre: Skipped 1 previous similar message [ 8541.797868] Lustre: lustre-MDT0001: new disk, initializing [ 8542.008875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8542.040237] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8548.072204] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8560.403122] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8568.462701] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8577.789848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8578.170859] Lustre: lustre-OST0000: new disk, initializing [ 8578.179480] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8578.206038] Lustre: 213717:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8580.262611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8580.278784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8580.523934] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8585.785667] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8595.344784] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8600.936771] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8601.173454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8605.712816] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8605.951435] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8611.299260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8611.308392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8611.313071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8611.313081] Lustre: Skipped 4 previous similar messages [ 8611.314412] LustreError: Skipped 1 previous similar message [ 8611.344580] Lustre: Skipped 4 previous similar messages [ 8614.245378] Lustre: server umount lustre-OST0000 complete [ 8624.599440] Lustre: server umount lustre-MDT0000 complete [ 8626.656689] LustreError: 211756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8626.689200] LustreError: 211756:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 8629.418982] LustreError: 212541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471587 with bad export cookie 8575769390248014292 [ 8629.422902] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8629.429908] LustreError: 212541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8630.041585] Lustre: server umount lustre-MDT0001 complete [ 8635.583178] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 17:39:50 (1788471590) [ 8653.974287] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 8665.228227] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 8710.191379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8710.524489] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8710.559927] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8710.697962] Lustre: lustre-MDT0000: new disk, initializing [ 8710.854340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8716.414104] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8729.114734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8729.245720] Lustre: 218328:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8729.447952] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8734.481506] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8745.500752] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8752.080996] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8760.247284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8760.568517] Lustre: 220298:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8762.448213] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8762.586618] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8767.227390] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8776.361332] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8781.291578] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8781.638241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8786.927329] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8787.273649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8793.573662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8793.573818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8793.594392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8793.594400] Lustre: Skipped 4 previous similar messages [ 8793.602964] Lustre: Skipped 3 previous similar messages [ 8793.637673] LustreError: Skipped 1 previous similar message [ 8796.381824] Lustre: server umount lustre-OST0000 complete [ 8808.001446] Lustre: server umount lustre-MDT0000 complete [ 8808.944831] LustreError: 218336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8808.987604] LustreError: 218336:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8813.066338] LustreError: 218322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471770 with bad export cookie 8575769390248015174 [ 8813.070348] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8813.090420] LustreError: 218322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8813.429000] Lustre: server umount lustre-MDT0001 complete [ 8823.567802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8824.060345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8824.075476] Lustre: Skipped 5 previous similar messages [ 8829.028194] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8839.417041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8844.865973] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8853.991668] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8860.389076] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8868.130754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8875.987239] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8884.691229] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8899.060584] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 8899.069850] LustreError: Skipped 1 previous similar message [ 8903.026147] Lustre: server umount lustre-OST0000 complete [ 8913.479910] Lustre: server umount lustre-MDT0000 complete [ 8917.581994] LustreError: 222494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471875 with bad export cookie 8575769390248016056 [ 8917.597111] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8917.599318] LustreError: 222494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8918.150678] Lustre: server umount lustre-MDT0001 complete [ 9026.545487] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 9030.226382] Key type lgssc unregistered [ 9030.650614] LNet: 226178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9030.660957] LNetError: 226178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9030.694462] LNet: Removed LNI 192.168.204.134@tcp [ 9031.938450] Key type .llcrypt unregistered [ 9031.944164] Key type ._llcrypt unregistered [ 9063.288105] Key type ._llcrypt registered [ 9063.290647] Key type .llcrypt registered [ 9063.392378] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 9080.638594] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 9082.004696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9082.055036] alg: No test for adler32 (adler32-zlib) [ 9083.124561] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 9083.425187] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 9085.127192] Key type lgssc registered [ 9086.468257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9128.131864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9128.167501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9129.781491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9129.849914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9129.947117] Lustre: lustre-MDT0000: new disk, initializing [ 9130.058823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9130.083390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9136.052149] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9149.036692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9149.275451] Lustre: 229529:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9149.405484] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9149.412635] Lustre: Skipped 1 previous similar message [ 9149.479581] Lustre: lustre-MDT0001: new disk, initializing [ 9149.591599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9149.657758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9149.685665] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9154.615390] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9163.929624] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9170.880791] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9180.164487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9180.493922] Lustre: lustre-OST0000: new disk, initializing [ 9180.501137] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9180.515645] Lustre: 231497:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9180.708779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9190.048044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9190.078319] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9190.417879] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9191.119939] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9203.168070] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9208.849237] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9209.069275] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9213.955951] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9214.153466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9221.090389] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9221.096783] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9221.104447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9221.151528] Lustre: Skipped 1 previous similar message [ 9226.219366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9226.232256] Lustre: Skipped 2 previous similar messages [ 9230.303148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9230.507273] Lustre: server umount lustre-OST0000 complete [ 9236.452498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9236.456429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9236.470592] Lustre: Skipped 1 previous similar message [ 9236.481035] Lustre: Skipped 1 previous similar message [ 9241.569031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9241.580710] Lustre: Skipped 1 previous similar message [ 9246.696104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9246.705756] Lustre: Skipped 1 previous similar message [ 9249.247670] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9249.468108] Lustre: server umount lustre-MDT0000 complete [ 9253.568227] LustreError: 229519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472211 with bad export cookie 16879232809080980026 [ 9253.580481] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9253.581330] LustreError: 229519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9254.003670] Lustre: server umount lustre-MDT0001 complete [ 9274.097958] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 9286.106682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9286.379123] LustreError: 234437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9286.406620] LustreError: 234437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9286.482801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9291.759303] LustreError: 234438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9291.790890] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9296.867174] LustreError: 234437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9301.198858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9301.448935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9305.315567] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9308.866560] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9316.680893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9317.138912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9325.314506] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9335.537653] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9335.729640] Lustre: lustre-OST0001: new disk, initializing [ 9335.733385] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9335.739240] Lustre: 236747:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9335.841310] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9342.201963] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9346.205777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9346.221417] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9346.329744] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9354.268800] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9363.201848] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9377.161414] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 17:52:12 (1788472332) [ 9382.375740] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9382.383614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9382.388375] Lustre: Skipped 3 previous similar messages [ 9382.404459] Lustre: Skipped 3 previous similar messages [ 9385.376245] Lustre: server umount lustre-MDT0000 complete [ 9387.501991] LustreError: 234432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9387.515271] LustreError: 234432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9389.389828] LustreError: 234418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472347 with bad export cookie 16879232809080980908 [ 9389.395096] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9389.409228] LustreError: 234418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9389.790500] Lustre: server umount lustre-MDT0001 complete [ 9403.818460] Lustre: server umount lustre-OST0000 complete [ 9417.829220] Lustre: server umount lustre-OST0001 complete [ 9425.683714] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 9437.447367] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 9477.328328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9477.803677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9477.863126] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9478.020102] Lustre: lustre-MDT0000: new disk, initializing [ 9478.290231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9478.343240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9483.379869] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9495.092099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9495.193637] Lustre: 240691:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9495.256450] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9495.260383] Lustre: Skipped 1 previous similar message [ 9495.314267] Lustre: lustre-MDT0001: new disk, initializing [ 9495.442098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9495.453247] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9500.075939] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9511.707967] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9518.427984] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9525.206102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9525.581716] Lustre: lustre-OST0000: new disk, initializing [ 9525.592770] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9525.604242] Lustre: 242661:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9527.144506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9527.156367] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9527.266866] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9532.159148] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9542.341734] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9546.906721] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9547.087771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9552.911490] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9553.134631] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9557.995125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9557.996155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9558.001315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9558.011172] LustreError: Skipped 2 previous similar messages [ 9558.055611] Lustre: Skipped 1 previous similar message [ 9569.247163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9569.431671] Lustre: server umount lustre-OST0000 complete [ 9577.451442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9577.462482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9579.735586] Lustre: server umount lustre-MDT0000 complete [ 9583.585525] LustreError: 240699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9583.614405] LustreError: 240699:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9583.766388] LustreError: 241487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472541 with bad export cookie 16879232809080983043 [ 9583.775522] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9583.777830] LustreError: 241487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9584.235945] Lustre: server umount lustre-MDT0001 complete [ 9593.300885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9593.905629] LustreError: 244880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9593.968325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9593.976183] Lustre: Skipped 2 previous similar messages [ 9598.833265] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9607.827237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9613.441539] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9621.853868] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9627.723810] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9635.225525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9642.621489] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9651.054844] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9797.602210] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9797.617524] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9797.629749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9797.629758] Lustre: Skipped 8 previous similar messages [ 9797.632258] Lustre: Skipped 2 previous similar messages [ 9801.536859] Lustre: server umount lustre-OST0000 complete [ 9807.840364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9812.278437] Lustre: server umount lustre-MDT0000 complete [ 9812.961659] LustreError: 247132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9812.976603] LustreError: 247132:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9816.195632] LustreError: 244861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472774 with bad export cookie 16879232809080983925 [ 9816.205629] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9816.219597] LustreError: 244861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9816.674438] Lustre: server umount lustre-MDT0001 complete [ 9921.266602] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 9924.981441] Key type lgssc unregistered [ 9925.339827] LNet: 248498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9925.356928] LNetError: 248498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9925.378358] LNet: Removed LNI 192.168.204.134@tcp [ 9926.420675] Key type .llcrypt unregistered [ 9926.426338] Key type ._llcrypt unregistered [ 9953.304376] Key type ._llcrypt registered [ 9953.306899] Key type .llcrypt registered [ 9953.407392] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 9970.158324] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 9971.612745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9971.659632] alg: No test for adler32 (adler32-zlib) [ 9972.761870] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 9973.080394] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 9974.831705] Key type lgssc registered [ 9976.028246] Lustre: Echo OBD driver; http://www.lustre.org/ [10026.317732] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [10037.129744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10037.152299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10038.484844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10038.565416] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10038.651328] Lustre: lustre-MDT0000: new disk, initializing [10038.715222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10038.727741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10043.684640] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10055.974894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10056.080622] Lustre: 252565:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10056.132912] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10056.137264] Lustre: Skipped 1 previous similar message [10056.235477] Lustre: lustre-MDT0001: new disk, initializing [10056.362859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10056.456685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10056.478753] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10061.876352] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10068.240206] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10075.638585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10075.940104] Lustre: lustre-OST0000: new disk, initializing [10075.944205] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10075.955714] Lustre: 254197:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10076.059214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10080.355658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10080.370798] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10080.480513] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10082.763420] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10092.897536] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10093.130202] Lustre: lustre-OST0001: new disk, initializing [10093.133688] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10093.142261] Lustre: 255070:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10093.246093] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10100.785563] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10101.865217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10101.878656] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10102.022792] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10113.542731] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10124.707084] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10138.086303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10138.104803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10138.119192] Lustre: Skipped 2 previous similar messages [10138.144876] Lustre: Skipped 3 previous similar messages [10143.222159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10143.229867] Lustre: Skipped 3 previous similar messages [10143.896330] Lustre: server umount lustre-MDT0000 complete [10147.958078] LustreError: 255838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473105 with bad export cookie 307013381916749299 [10147.980197] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10148.387190] Lustre: server umount lustre-MDT0001 complete [10162.888418] Lustre: server umount lustre-OST0000 complete [10164.514521] Lustre: 250071:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788473106/real 1788473106] req@ffff92dd7800c000 x1875349791128320/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788473122 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10164.555799] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10164.569372] Lustre: Skipped 1 previous similar message [10167.085604] Lustre: server umount lustre-OST0001 complete [10181.772037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10193.294946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10203.039042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10213.828898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10236.119987] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [10245.533966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10245.557742] Lustre: lustre-MDT0000: reset Object Index mappings [10246.037476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10250.147471] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10251.242514] LustreError: 260084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10251.267187] LustreError: 260084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10255.350397] LustreError: 260085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10259.372328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10259.395440] Lustre: lustre-MDT0001: reset Object Index mappings [10259.943583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10264.809710] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10268.827267] Lustre: 261228:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10276.795571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10277.214986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10284.393659] LustreError: 261585:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10284.535922] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10289.641234] LustreError: 261583:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10290.696294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10293.953824] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10299.392525] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10301.640113] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10310.347494] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10315.542675] Lustre: 263101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10324.966568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10324.976192] Lustre: Skipped 2 previous similar messages [10324.981180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10325.015244] Lustre: Skipped 3 previous similar messages [10328.730081] Lustre: server umount lustre-MDT0000 complete [10330.085298] LustreError: 260817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10330.114779] LustreError: 260817:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10332.609950] LustreError: 260066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473290 with bad export cookie 307013381916756089 [10332.613150] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10332.625805] LustreError: 260066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10333.395910] Lustre: server umount lustre-MDT0001 complete [10348.790914] Lustre: server umount lustre-OST0000 complete [10351.583900] Lustre: 250073:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788473293/real 1788473293] req@ffff92dd71fb5880 x1875349791181952/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788473309 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10351.625849] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10351.642638] Lustre: Skipped 1 previous similar message [10352.975062] Lustre: server umount lustre-OST0001 complete [10371.976551] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 18:08:47 (1788473327) [10390.671934] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [10403.683166] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [10443.224946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10443.429719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10443.452742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10443.518551] Lustre: lustre-MDT0000: new disk, initializing [10443.613485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10443.627325] Lustre: Skipped 1 previous similar message [10443.653144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10448.152735] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10461.154359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10461.400478] Lustre: 267354:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10461.479729] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10461.484449] Lustre: Skipped 1 previous similar message [10461.576699] Lustre: lustre-MDT0001: new disk, initializing [10461.746869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10461.757453] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10466.907821] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10478.201102] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10485.047399] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10492.587422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10492.925906] Lustre: lustre-OST0000: new disk, initializing [10492.932156] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10492.942236] Lustre: 269323:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10494.394607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10494.415413] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10494.556144] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10500.465311] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10513.014680] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10519.604554] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10519.882351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10525.298899] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10525.795623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10530.272563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10530.278705] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10530.282168] LustreError: Skipped 1 previous similar message [10530.284948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10530.304219] Lustre: Skipped 1 previous similar message [10530.330376] Lustre: Skipped 1 previous similar message [10534.213598] Lustre: server umount lustre-OST0000 complete [10538.468453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10538.485547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10538.510317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10544.763576] Lustre: server umount lustre-MDT0000 complete [10545.634087] LustreError: 267365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10545.645094] LustreError: 267365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10548.920856] LustreError: 267345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473506 with bad export cookie 307013381916758189 [10548.944663] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10549.504896] Lustre: server umount lustre-MDT0001 complete [10574.068306] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [10585.711930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10586.363446] LustreError: 272271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10586.432312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10586.444404] Lustre: Skipped 2 previous similar messages [10591.754541] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10600.953095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10606.221227] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10609.381281] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10616.855694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10625.315949] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10637.158600] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10637.372887] Lustre: lustre-OST0001: new disk, initializing [10637.388152] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10637.395853] Lustre: 274583:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10639.473813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10639.503835] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10639.646658] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10646.411467] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10658.943772] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10663.571273] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10670.589766] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10670.654327] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10683.361338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10683.382810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10683.401340] Lustre: Skipped 1 previous similar message [10683.407219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10683.411419] Lustre: Skipped 2 previous similar messages [10688.146635] Lustre: server umount lustre-MDT0000 complete [10691.058877] LustreError: 272268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10691.106821] LustreError: 272268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10692.641518] LustreError: 272254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473650 with bad export cookie 307013381916759071 [10692.657169] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10692.680535] LustreError: 272254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10693.396154] Lustre: server umount lustre-MDT0001 complete [10708.357512] Lustre: server umount lustre-OST0000 complete [10713.068413] Lustre: 250073:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788473654/real 1788473654] req@ffff92dd77319c00 x1875349791302144/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788473670 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10713.597638] Lustre: server umount lustre-OST0001 complete [10734.342180] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [10747.416906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10748.129431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10748.138423] Lustre: Skipped 3 previous similar messages [10753.802338] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10757.633266] LustreError: 277931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10757.656939] LustreError: 277931:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10764.178319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10770.283614] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10774.603994] Lustre: 279073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10783.240596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10791.311364] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10793.021789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10801.961209] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10807.819448] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10810.493566] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10820.029690] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10824.162707] Lustre: 280951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10831.115747] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10831.198231] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10845.818809] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 18:16:41 (1788473801) [10862.049581] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10862.055882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10862.080025] Lustre: Skipped 4 previous similar messages [10862.092267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10862.099026] Lustre: Skipped 4 previous similar messages [10876.383841] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10876.574026] Lustre: server umount lustre-MDT0000 complete [10881.119992] LustreError: 277913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473838 with bad export cookie 307013381916765938 [10881.126818] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10881.134727] LustreError: 277913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10886.629760] LustreError: 280913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10886.655123] LustreError: 280913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [10887.552425] Lustre: server umount lustre-MDT0001 complete [10906.080098] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10906.235567] Lustre: server umount lustre-OST0000 complete [10910.930048] Lustre: server umount lustre-OST0001 complete [10917.993499] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [10929.884554] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [10981.996924] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [10993.676776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10993.927278] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10993.958623] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10994.033383] Lustre: lustre-MDT0000: new disk, initializing [10994.137756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11000.027084] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11011.237934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11011.490917] Lustre: 285485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11011.945753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11011.957167] Lustre: Skipped 4 previous similar messages [11012.032783] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11017.077458] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11029.652277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11029.858418] Lustre: 287120:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11032.080990] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11038.546774] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11052.164170] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11052.569133] Lustre: 287991:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11054.077620] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11054.087516] Lustre: Skipped 1 previous similar message [11054.318854] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11060.597858] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11071.743780] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11075.813603] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11075.821983] Lustre: Skipped 1 previous similar message [11091.133215] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 18:20:47 (1788474047) [11093.997718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11094.005870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11094.023339] Lustre: Skipped 5 previous similar messages [11094.035388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11094.043326] Lustre: Skipped 16 previous similar messages [11095.591036] Lustre: server umount lustre-MDT0000 complete [11100.103309] LustreError: 285478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474057 with bad export cookie 307013381916773428 [11100.145655] LustreError: 285478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11100.156286] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11100.751522] Lustre: server umount lustre-MDT0001 complete [11115.861757] Lustre: server umount lustre-OST0000 complete [11117.027695] Lustre: 250071:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788474058/real 1788474058] req@ffff92dd76b8e680 x1875349791464704/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788474074 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11119.275605] Lustre: server umount lustre-OST0001 complete [11128.706247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11134.942175] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11144.152710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11149.239558] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11156.942455] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11162.146966] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11169.164900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11169.262655] LustreError: 292071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11169.284665] LustreError: 292071:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [11175.992618] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11184.067787] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11200.881816] loop1: detected capacity change from 0 to 204800000 [11201.200454] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11201.213570] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11201.223213] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11206.695831] loop1: detected capacity change from 0 to 204800000 [11213.244465] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11213.458643] LustreError: 290230:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs2622: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11213.478656] LustreError: 290230:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs2622 [11213.488708] LustreError: 290230:0:(mgs_handler.c:517:mgs_target_reg()) fs2622-MDT0001: Error checking target index: rc = -13 [11213.501466] LustreError: fs2622-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11213.522456] LustreError: 293959:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11213.529929] LustreError: 293959:0:(tgt_mount.c:2132:server_put_super()) no obd fs2622-MDT0001 [11213.537632] LustreError: 293959:0:(tgt_mount.c:116:server_deregister_mount()) fs2622-MDT0001 not registered [11213.600028] Lustre: server umount fs2622-MDT0001 complete [11213.602910] LustreError: 293959:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11223.783937] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11224.008539] Lustre: Setting parameter fs2622-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs2622-MDT0001 [11224.072431] Lustre: srv-fs2622-MDT0001: No data found on store. Initialize space. [11224.079422] Lustre: Skipped 4 previous similar messages [11224.257632] Lustre: fs2622-MDT0001: new disk, initializing [11224.263574] Lustre: Skipped 3 previous similar messages [11224.440186] Lustre: srv-fs2622-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11224.453205] Lustre: 294425:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2622-MDT0001-mdtlov: can't create new llog: rc = -115 [11224.483132] LustreError: 294425:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2622-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11230.718429] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11238.722045] Lustre: server umount fs2622-MDT0001 complete [11254.505970] loop1: detected capacity change from 0 to 204800000 [11256.822868] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11261.703234] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11262.312552] Lustre: 295994:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2622-MDT0001-mdtlov: can't create new llog: rc = -115 [11262.321401] LustreError: 295994:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2622-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11267.520931] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11273.002597] Lustre: server umount fs2622-MDT0001 complete [11287.134058] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.34@tcp (stopping) [11287.531885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11287.557890] Lustre: Skipped 5 previous similar messages [11299.297310] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11299.539351] Lustre: server umount lustre-MDT0000 complete [11307.867384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11308.008891] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11308.020362] LustreError: 250070:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92dd7c556a00 x1875349791539584/t0(0) o250->MGC192.168.204.134@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'ptlrpcd_00_02.0' uid:0 gid:0 projid:4294967295 [11312.516401] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11313.651218] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11321.104437] loop1: detected capacity change from 0 to 204800000 [11326.801126] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11327.217386] Lustre: 298444:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2622-MDT0001-mdtlov: can't create new llog: rc = -115 [11327.235611] LustreError: 298444:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2622-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11332.389087] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11337.253377] Lustre: server umount fs2622-MDT0001 complete [11364.321055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11364.325906] LustreError: Skipped 1 previous similar message [11370.502785] Lustre: server umount lustre-OST0000 complete [11381.677097] Lustre: server umount lustre-MDT0000 complete [11387.314178] LustreError: 290222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474345 with bad export cookie 307013381916776109 [11387.328461] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11387.336703] LustreError: 290222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11388.000305] Lustre: server umount lustre-MDT0001 complete [11406.842273] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [11410.236743] Key type lgssc unregistered [11410.602246] LNet: 300811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11410.615754] LNetError: 300811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11410.634063] LNet: Removed LNI 192.168.204.134@tcp [11411.451592] Key type .llcrypt unregistered [11411.454189] Key type ._llcrypt unregistered [11433.124387] Key type ._llcrypt registered [11433.126594] Key type .llcrypt registered [11433.247861] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [11434.710099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11434.738949] alg: No test for adler32 (adler32-zlib) [11436.097506] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [11436.402369] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [11438.063219] Key type lgssc registered [11439.080710] Lustre: Echo OBD driver; http://www.lustre.org/ [11446.898467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11446.925683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11448.457117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11452.853190] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11459.044410] LustreError: 302082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11459.071441] LustreError: 302082:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11461.489396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11461.935520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11466.380604] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11474.268624] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11480.260099] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11488.542477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11489.174609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11493.223086] LustreError: 303911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11496.453181] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11498.469838] LustreError: 303910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11503.587094] LustreError: 304111:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11504.889739] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11507.632325] LustreError: 303911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11507.679830] LustreError: 303911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11516.899137] LustreError: 303909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11516.946944] LustreError: 303909:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11526.348789] loop1: detected capacity change from 0 to 204800000 [11526.631275] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11526.644627] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11526.659201] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11532.812522] loop1: detected capacity change from 0 to 204800000 [11533.346089] LustreError: 303909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11533.376199] LustreError: 303909:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [11539.729671] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11539.928410] Lustre: fs11000-OST0000: new disk, initializing [11539.932147] Lustre: srv-fs11000-OST0000: No data found on store. Initialize space. [11539.946541] Lustre: 305817:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11540.098252] Lustre: fs11000-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11548.811705] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11558.981865] Lustre: server umount fs11000-OST0000 complete [11566.056113] LustreError: 303910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11566.086364] LustreError: 303910:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [11571.178961] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11572.997883] loop1: detected capacity change from 0 to 204800000 [11579.456643] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11579.830917] Lustre: fs11000-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11586.548721] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11587.045597] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11597.164043] loop2: detected capacity change from 0 to 204800000 [11597.446413] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11597.454992] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11597.467301] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11603.142199] loop2: detected capacity change from 0 to 204800000 [11608.449153] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11608.542183] LustreError: fs11000-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11608.550737] LustreError: fs11000-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11608.560162] LustreError: 308861:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11608.566332] LustreError: 308861:0:(tgt_mount.c:2132:server_put_super()) no obd fs11000-OST0001 [11608.570894] LustreError: 308861:0:(tgt_mount.c:116:server_deregister_mount()) fs11000-OST0001 not registered [11608.609417] Lustre: server umount fs11000-OST0001 complete [11608.612422] LustreError: 308861:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11619.183408] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11619.312348] Lustre: fs11000-OST0001: new disk, initializing [11619.318298] Lustre: srv-fs11000-OST0001: No data found on store. Initialize space. [11619.323449] Lustre: 309413:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11619.383655] Lustre: fs11000-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11625.583472] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11630.630849] LustreError: 307132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11630.648391] LustreError: 307132:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 36 previous similar messages [11634.028696] Lustre: server umount fs11000-OST0000 complete [11643.124450] Lustre: server umount fs11000-OST0001 complete [11655.063385] loop1: detected capacity change from 0 to 204800000 [11660.288739] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11660.535504] Lustre: fs11000-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11666.813764] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11673.410043] loop2: detected capacity change from 0 to 204800000 [11679.041875] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11685.501826] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11694.235719] Lustre: server umount fs11000-OST0000 complete [11704.635121] Lustre: server umount fs11000-OST0001 complete [11723.747835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11723.776361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11723.846140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11724.768477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11724.775117] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11728.513444] Lustre: server umount lustre-OST0000 complete [11735.011594] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11735.015645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11735.036051] Lustre: Skipped 1 previous similar message [11735.070319] Lustre: Skipped 2 previous similar messages [11740.129387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11740.146788] Lustre: Skipped 1 previous similar message [11744.226303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11744.236375] Lustre: Skipped 1 previous similar message [11748.319385] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11748.608287] Lustre: server umount lustre-MDT0000 complete [11752.559445] LustreError: 302061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474710 with bad export cookie 6350208500737716552 [11752.566373] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11752.574975] LustreError: 302061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11753.115684] Lustre: server umount lustre-MDT0001 complete [11765.033529] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [11768.047863] Key type lgssc unregistered [11768.326698] LNet: 315336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11768.333937] LNetError: 315336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11768.354613] LNet: Removed LNI 192.168.204.134@tcp [11769.242230] Key type .llcrypt unregistered [11769.247291] Key type ._llcrypt unregistered [11801.380126] Key type ._llcrypt registered [11801.385188] Key type .llcrypt registered [11801.551306] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [11816.810217] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [11817.684786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11817.820941] alg: No test for adler32 (adler32-zlib) [11818.896198] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [11819.123217] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [11820.839191] Key type lgssc registered [11821.938575] Lustre: Echo OBD driver; http://www.lustre.org/ [11859.983132] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11860.004434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11861.395771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11861.433083] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11861.570499] Lustre: lustre-MDT0000: new disk, initializing [11861.729272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11861.765592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11866.765593] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11878.633046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11878.757935] Lustre: 319195:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11878.830220] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11878.836762] Lustre: Skipped 1 previous similar message [11878.933830] Lustre: lustre-MDT0001: new disk, initializing [11879.039830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11879.072810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11879.091437] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11884.846500] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11895.121257] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11900.774287] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11909.579094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11910.052342] Lustre: lustre-OST0000: new disk, initializing [11910.055141] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11910.063233] Lustre: 321167:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11910.177213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11916.949559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11916.960870] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11917.213623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11920.704664] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11930.802547] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11933.097626] Lustre: Mounted lustre-client - version 2.17.58_2_gbb63b80 [11934.238669] Lustre: Unmounted lustre-client [11942.880723] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11942.886046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11942.895347] LustreError: Skipped 1 previous similar message [11942.897093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11942.918357] Lustre: Skipped 1 previous similar message [11942.949739] Lustre: Skipped 1 previous similar message [11943.399802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11943.407777] Lustre: Skipped 1 previous similar message [11948.516822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11948.526908] Lustre: Skipped 1 previous similar message [11948.873729] Lustre: server umount lustre-OST0000 complete [11953.633892] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11953.648266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11953.652131] Lustre: Skipped 1 previous similar message [11958.759307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11958.780609] Lustre: Skipped 1 previous similar message [11959.003723] Lustre: server umount lustre-MDT0000 complete [11963.113033] LustreError: 319186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474920 with bad export cookie 3376854354271561909 [11963.119982] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11963.121133] LustreError: 319186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11963.468907] Lustre: server umount lustre-MDT0001 complete [11974.278801] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [11977.018913] Key type lgssc unregistered [11988.127370] LNet: 1 local NIs in recovery (showing 1): 192.168.204.134@tcp [11988.515579] LNet: 323327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11988.532051] LNetError: 323327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12064.360846] LNet: Removed LNI 192.168.204.134@tcp [12065.427342] Key type .llcrypt unregistered [12065.435915] Key type ._llcrypt unregistered [12084.842463] Key type ._llcrypt registered [12084.848874] Key type .llcrypt registered [12084.991756] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [12103.571773] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [12104.614490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12104.694142] alg: No test for adler32 (adler32-zlib) [12106.729212] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [12107.107913] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [12108.783263] Key type lgssc registered [12110.498915] Lustre: Echo OBD driver; http://www.lustre.org/ [12150.491610] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 18:38:26 (1788475106) [12158.453236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12158.496683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12159.682528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12159.703609] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12159.754995] Lustre: lustre-MDT0000: new disk, initializing [12159.820482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12159.842753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12165.307130] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [12176.051506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12176.189421] Lustre: 326546:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12176.251317] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12176.255762] Lustre: Skipped 1 previous similar message [12176.345720] Lustre: lustre-MDT0001: new disk, initializing [12176.488917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12176.547371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12176.571356] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12180.979575] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [12192.568783] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12198.576942] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12206.015940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12206.338237] Lustre: lustre-OST0000: new disk, initializing [12206.342246] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12206.357198] Lustre: 328514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12206.487833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12213.855939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12213.871064] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12214.034476] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12214.423376] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [12225.336662] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12244.960038] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12244.981459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12244.985979] LustreError: Skipped 1 previous similar message [12244.995232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12245.013613] Lustre: Skipped 1 previous similar message [12245.045927] Lustre: Skipped 1 previous similar message [12247.646152] Lustre: server umount lustre-OST0000 complete [12253.154147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12253.174088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12253.203984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12255.201113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12255.202666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12255.249154] Lustre: Skipped 1 previous similar message [12258.760234] Lustre: server umount lustre-MDT0000 complete [12263.021082] LustreError: 326540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788475220 with bad export cookie 7780293756458708384 [12263.024377] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12263.031551] LustreError: 326540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12263.402501] Lustre: server umount lustre-MDT0001 complete [12276.916303] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [12281.168786] Key type lgssc unregistered [12281.601735] LNet: 330423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12281.612131] LNetError: 330423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12281.636229] LNet: Removed LNI 192.168.204.134@tcp [12282.879872] Key type .llcrypt unregistered [12282.881504] Key type ._llcrypt unregistered [12304.385306] Key type ._llcrypt registered [12304.388276] Key type .llcrypt registered [12304.473855] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [12305.532513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12305.869897] alg: No test for adler32 (adler32-zlib) [12306.940158] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [12307.186914] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [12308.871247] Key type lgssc registered [12310.083506] Lustre: Echo OBD driver; http://www.lustre.org/ [12318.967694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12318.993397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12320.698334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12325.364978] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [12330.977086] LustreError: 331693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12330.993328] LustreError: 331693:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12335.402865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12336.218691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12342.051955] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [12349.758186] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12355.852588] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12362.830304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12363.211282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12369.821641] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [12377.834880] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12391.919840] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12393.441097] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12397.536749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12397.548257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12397.560680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12399.009743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12399.015835] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12399.027562] Lustre: Skipped 1 previous similar message [12399.061605] Lustre: Skipped 1 previous similar message [12401.203504] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.34@tcp (stopping) [12401.218391] Lustre: Skipped 1 previous similar message [12403.484362] Lustre: server umount lustre-MDT0000 complete [12404.204597] LustreError: 334189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12404.228472] LustreError: 334189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12406.306694] LustreError: 331689:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12407.481325] LustreError: 331674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788475365 with bad export cookie 5212788136761981003 [12407.490149] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12407.496803] LustreError: 331674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12409.312659] LustreError: 334189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12409.321957] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12409.339647] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12414.025313] Lustre: server umount lustre-MDT0001 complete [12418.403772] Lustre: server umount lustre-OST0000 complete [12428.519361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12433.500876] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [12436.217588] Lustre: 335877:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.134@tcp [12438.506513] Lustre: server umount MGS complete [12446.825632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12447.341755] LustreError: 336450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12447.365297] LustreError: 336450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12447.414144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12452.051850] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [12456.487307] LustreError: 336447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12456.519048] LustreError: 336447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12460.359334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12460.783209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12466.159267] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [12475.421520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12475.876154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12483.192290] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [12490.823255] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12492.551870] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12498.794751] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12500.480936] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12506.595288] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12506.618730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12506.633252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12506.642804] Lustre: Skipped 2 previous similar messages [12507.527627] Lustre: server umount lustre-OST0000 complete [12512.224736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12512.235198] LustreError: Skipped 1 previous similar message [12512.246109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12512.258993] Lustre: Skipped 1 previous similar message [12517.856798] LustreError: 336451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12517.889510] LustreError: 336451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12518.060233] Lustre: server umount lustre-MDT0000 complete [12522.440501] LustreError: 336432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788475480 with bad export cookie 5212788136761981836 [12522.456707] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12522.459152] LustreError: 336432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12522.727535] Lustre: server umount lustre-MDT0001 complete [12534.825359] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [12537.700532] Key type lgssc unregistered [12538.071507] LNet: 339887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12538.078472] LNetError: 339887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12538.108412] LNet: Removed LNI 192.168.204.134@tcp [12539.166878] Key type .llcrypt unregistered [12539.173125] Key type ._llcrypt unregistered [12562.917619] Key type ._llcrypt registered [12562.921553] Key type .llcrypt registered [12563.071756] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [12580.420334] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [12581.576774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12581.619420] alg: No test for adler32 (adler32-zlib) [12582.823267] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [12583.001101] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [12584.679140] Key type lgssc registered [12585.603812] Lustre: Echo OBD driver; http://www.lustre.org/ [12623.952987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12623.976805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12625.258566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12625.322561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12625.444581] Lustre: lustre-MDT0000: new disk, initializing [12625.582628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12625.598665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12631.308299] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [12642.973843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12643.106524] Lustre: 343049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12643.185084] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12643.193180] Lustre: Skipped 1 previous similar message [12643.313609] Lustre: lustre-MDT0001: new disk, initializing [12643.514948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12643.552586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12643.564166] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12648.387188] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [12658.904430] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12665.035575] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12671.778324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12672.018204] Lustre: lustre-OST0000: new disk, initializing [12672.021735] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12672.029673] Lustre: 345015:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12672.140839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12678.757792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12678.774687] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12678.966315] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12679.664586] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [12690.645890] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12696.768328] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12697.107582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12703.938732] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12704.156747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12709.869809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12709.873149] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12709.908410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12709.919344] Lustre: Skipped 1 previous similar message [12712.635658] Lustre: server umount lustre-OST0000 complete [12720.101655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12720.115639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12720.115651] Lustre: Skipped 1 previous similar message [12720.118475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12720.132719] LustreError: Skipped 1 previous similar message [12720.149807] Lustre: Skipped 1 previous similar message [12722.574984] Lustre: server umount lustre-MDT0000 complete [12726.536896] LustreError: 343041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788475684 with bad export cookie 2489209670692969359 [12726.538611] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12726.545278] LustreError: 343041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12727.006770] Lustre: server umount lustre-MDT0001 complete [12736.220167] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 18:48:11 (1788475691) [12756.398504] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [12771.233204] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [12813.003272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12813.299521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12813.343745] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12813.463834] Lustre: lustre-MDT0000: new disk, initializing [12813.596127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12813.617835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12818.846846] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [12829.680981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12829.797418] Lustre: 349790:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12829.846796] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12829.858809] Lustre: Skipped 1 previous similar message [12829.950766] Lustre: lustre-MDT0001: new disk, initializing [12830.072406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12830.113631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12830.119945] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12834.697727] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [12845.016914] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12851.814340] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12860.624165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12861.105706] Lustre: lustre-OST0000: new disk, initializing [12861.113333] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12861.137108] Lustre: 351760:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12861.272708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12868.024624] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [12868.735733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12868.756229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12868.965520] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12878.483874] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12920.302952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12920.316948] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12920.321953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12920.336072] Lustre: Skipped 2 previous similar messages [12920.383423] LustreError: Skipped 1 previous similar message [12923.445265] Lustre: server umount lustre-OST0000 complete [12928.730749] Lustre: server umount lustre-MDT0000 complete [12930.531596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12930.562600] LustreError: 349796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12930.595827] LustreError: 349796:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12934.619576] LustreError: 349784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788475892 with bad export cookie 2489209670692970234 [12934.629429] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12934.650638] LustreError: 349784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12935.172756] Lustre: server umount lustre-MDT0001 complete [12950.305398] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [12953.455538] Key type lgssc unregistered [12953.802725] LNet: 353936:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12953.810359] LNetError: 353936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12953.838983] LNet: Removed LNI 192.168.204.134@tcp [12954.997866] Key type .llcrypt unregistered [12955.002681] Key type ._llcrypt unregistered [13063.306237] Key type ._llcrypt registered [13063.308693] Key type .llcrypt registered [13063.408465] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [13063.741285] Key type .llcrypt unregistered [13063.748098] Key type ._llcrypt unregistered [13152.332509] Key type ._llcrypt registered [13152.334023] Key type .llcrypt registered [13152.468148] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [13152.929365] Key type .llcrypt unregistered [13152.932322] Key type ._llcrypt unregistered [13188.471504] Key type ._llcrypt registered [13188.477162] Key type .llcrypt registered [13188.610489] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [13189.080368] Key type .llcrypt unregistered [13189.082447] Key type ._llcrypt unregistered [13222.408560] Key type ._llcrypt registered [13222.412536] Key type .llcrypt registered [13222.582954] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [13223.016490] Key type .llcrypt unregistered [13223.020607] Key type ._llcrypt unregistered [13310.111749] Key type ._llcrypt registered [13310.115222] Key type .llcrypt registered [13310.280604] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [13310.600831] Key type .llcrypt unregistered [13310.603598] Key type ._llcrypt unregistered [13344.997802] Key type ._llcrypt registered [13345.000959] Key type .llcrypt registered [13345.203378] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [13345.588313] Key type .llcrypt unregistered [13345.591423] Key type ._llcrypt unregistered [13382.008451] Key type ._llcrypt registered [13382.010972] Key type .llcrypt registered [13382.157199] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [13382.503351] Key type .llcrypt unregistered [13382.507166] Key type ._llcrypt unregistered [13416.869703] Key type ._llcrypt registered [13416.871798] Key type .llcrypt registered [13416.984330] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [13417.198366] Key type .llcrypt unregistered [13417.204217] Key type ._llcrypt unregistered [13452.490120] Key type ._llcrypt registered [13452.492219] Key type .llcrypt registered [13452.616890] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [13452.924437] Key type .llcrypt unregistered [13452.927104] Key type ._llcrypt unregistered [13488.637390] Key type ._llcrypt registered [13488.640498] Key type .llcrypt registered [13488.778140] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [13489.140319] Key type .llcrypt unregistered [13489.142924] Key type ._llcrypt unregistered [13555.752738] Key type ._llcrypt registered [13555.760631] Key type .llcrypt registered [13555.937560] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [13556.311495] Key type .llcrypt unregistered [13556.316384] Key type ._llcrypt unregistered [13589.551547] Key type ._llcrypt registered [13589.554953] Key type .llcrypt registered [13589.679495] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [13590.000394] Key type .llcrypt unregistered [13590.003884] Key type ._llcrypt unregistered [13625.837768] Key type ._llcrypt registered [13625.840287] Key type .llcrypt registered [13625.980734] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [13626.431933] Key type .llcrypt unregistered [13626.437908] Key type ._llcrypt unregistered [13678.940750] Key type ._llcrypt registered [13678.948285] Key type .llcrypt registered [13679.240217] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [13695.903351] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [13697.204346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13697.224859] alg: No test for adler32 (adler32-zlib) [13698.454915] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [13698.777784] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [13700.423241] Key type lgssc registered [13701.850387] Lustre: Echo OBD driver; http://www.lustre.org/ [13744.647940] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13744.681536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13745.980394] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13745.999467] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13746.091692] Lustre: tst250fs-MDT0000: new disk, initializing [13746.175235] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13746.201589] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13751.816525] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [13763.573535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13763.786750] Lustre: 374252:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13763.846889] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13763.852077] Lustre: Skipped 1 previous similar message [13763.921114] Lustre: tst250fs-MDT0001: new disk, initializing [13763.996335] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13764.048384] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13764.060904] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13768.537886] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [13773.937837] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13782.040416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13782.318199] Lustre: tst250fs-OST0000: new disk, initializing [13782.325457] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13782.332352] Lustre: 375884:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13782.348876] Lustre: tst250fs-OST0000: Not available for connect from 0@lo (not set up) [13782.467705] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13787.719857] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13787.749080] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13787.791541] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [13789.695102] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [13801.155686] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13801.325747] Lustre: tst250fs-OST0001: new disk, initializing [13801.328565] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13801.333738] Lustre: 376758:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13801.426108] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13807.618905] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [13807.718053] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13807.732410] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13807.841541] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13813.644509] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13815.962629] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13815.968680] Lustre: Skipped 1 previous similar message [13838.823265] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13838.829599] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13838.873268] Lustre: Skipped 1 previous similar message [13838.929955] Lustre: Skipped 3 previous similar messages [13842.601860] Lustre: server umount tst250fs-MDT0000 complete [13847.322129] LustreError: 375056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476805 with bad export cookie 16764644195264417345 [13847.326443] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13847.337475] LustreError: 375056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13847.928659] Lustre: server umount tst250fs-MDT0001 complete [13862.945810] Lustre: server umount tst250fs-OST0000 complete [13864.931218] Lustre: 372473:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788476806/real 1788476806] req@ffff92dc42725500 x1875353698114304/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788476822 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13864.979259] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13865.001660] Lustre: Skipped 2 previous similar messages [13867.487125] Lustre: 372475:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788476809/real 1788476809] req@ffff92dc4d351180 x1875353698114560/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788476825 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13868.146841] Lustre: server umount tst250fs-OST0001 complete [13889.399653] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [13899.657790] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [13936.652958] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 19:08:12 (1788476892) [13938.532426] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13957.083587] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [13972.058501] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [14008.899807] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13674 sec ========= 19:09:24 (1788476964) [14010.888948] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:09:26 (1788476966) === [14014.457476] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:09:30 (1788476970) === [14056.880442] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [14059.766564] Key type lgssc unregistered [14060.095722] LNet: 385907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14060.107716] LNetError: 385907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14060.129784] LNet: Removed LNI 192.168.204.134@tcp [14061.037292] Key type .llcrypt unregistered [14061.039093] Key type ._llcrypt unregistered