[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 475611448 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003099] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010146] LSM: Security Framework initializing [ 0.011054] Yama: becoming mindful. [ 0.012043] SELinux: Initializing. [ 0.014004] *** VALIDATE selinux *** [ 0.022325] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027825] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028179] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030125] *** VALIDATE tmpfs *** [ 0.032494] *** VALIDATE proc *** [ 0.034065] *** VALIDATE cgroup *** [ 0.035009] *** VALIDATE cgroup2 *** [ 0.036246] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039036] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.042977] debug: unmapping init [mem 0xffffffff9da59000-0xffffffff9da60fff] [ 0.044924] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045783] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046023] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050017] ... max period: 00007fffffffffff [ 0.051014] ... fixed-purpose events: 3 [ 0.052010] ... event mask: 000000070000000f [ 0.053301] rcu: Hierarchical SRCU implementation. [ 0.055493] smp: Bringing up secondary CPUs ... [ 0.056612] x86: Booting SMP configuration: [ 0.057024] .... node #0, CPUs: #1 #2 #3 [ 0.064528] smp: Brought up 1 node, 4 CPUs [ 0.065955] smpboot: Max logical packages: 1 [ 0.066015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145644] node 0 deferred pages initialised in 78ms [ 0.149011] devtmpfs: initialized [ 0.150212] x86/mm: Memory block size: 128MB [ 0.152842] gcov: version magic: 0x41383552 [ 0.154155] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156229] pinctrl core: initialized pinctrl subsystem [ 0.157175] [ 0.157545] ************************************************************* [ 0.158010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159008] ** ** [ 0.160011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161011] ** ** [ 0.162013] ** This means that this kernel is built to expose internal ** [ 0.163012] ** IOMMU data structures, which may compromise security on ** [ 0.164010] ** your system. ** [ 0.165010] ** ** [ 0.166010] ** If you see this message and you are not debugging the ** [ 0.167010] ** kernel, report this immediately to your vendor! ** [ 0.168011] ** ** [ 0.169010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170010] ************************************************************* [ 0.171703] NET: Registered protocol family 16 [ 0.172414] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.173063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.174059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.175431] cpuidle: using governor menu [ 0.177894] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.179374] PCI: Using configuration type 1 for base access [ 0.180093] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189153] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.190017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.191173] cryptd: max_cpu_qlen set to 1000 [ 0.194290] ACPI: Added _OSI(Module Device) [ 0.196016] ACPI: Added _OSI(Processor Device) [ 0.197010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.198030] ACPI: Added _OSI(Processor Aggregator Device) [ 0.202084] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.207325] ACPI: Interpreter enabled [ 0.209060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.210011] ACPI: Using IOAPIC for interrupt routing [ 0.212105] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.215370] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.225588] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.228046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.231024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.235090] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.240553] acpiphp: Slot [2] registered [ 0.242135] acpiphp: Slot [5] registered [ 0.244137] acpiphp: Slot [6] registered [ 0.245247] acpiphp: Slot [7] registered [ 0.246145] acpiphp: Slot [8] registered [ 0.248142] acpiphp: Slot [9] registered [ 0.249132] acpiphp: Slot [10] registered [ 0.251121] acpiphp: Slot [3] registered [ 0.252228] acpiphp: Slot [4] registered [ 0.254115] acpiphp: Slot [11] registered [ 0.256144] acpiphp: Slot [12] registered [ 0.257099] acpiphp: Slot [13] registered [ 0.259124] acpiphp: Slot [14] registered [ 0.261105] acpiphp: Slot [15] registered [ 0.263115] acpiphp: Slot [16] registered [ 0.264104] acpiphp: Slot [17] registered [ 0.266139] acpiphp: Slot [18] registered [ 0.268100] acpiphp: Slot [19] registered [ 0.269131] acpiphp: Slot [20] registered [ 0.271126] acpiphp: Slot [21] registered [ 0.272117] acpiphp: Slot [22] registered [ 0.274122] acpiphp: Slot [23] registered [ 0.276146] acpiphp: Slot [24] registered [ 0.277096] acpiphp: Slot [25] registered [ 0.279109] acpiphp: Slot [26] registered [ 0.280109] acpiphp: Slot [27] registered [ 0.282110] acpiphp: Slot [28] registered [ 0.284135] acpiphp: Slot [29] registered [ 0.285116] acpiphp: Slot [30] registered [ 0.287136] acpiphp: Slot [31] registered [ 0.288059] PCI host bridge to bus 0000:00 [ 0.289019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.292023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.294029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.297052] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.300022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.302037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.304246] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.307065] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.310297] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.320813] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.325610] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.328017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.329018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.331011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.333495] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.336780] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.339042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.342832] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.349016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.363898] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.369016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.374781] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.539019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.554019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.586031] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.603646] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.614018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.628021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.685017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.725079] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.738019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.752023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.774019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.790927] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.800019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.809017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.829020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.841072] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.847018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.852016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.862016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.869013] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.874026] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.877016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.890016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.900604] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.903404] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.905380] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.908391] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.911227] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.916057] iommu: Default domain type: Passthrough [ 0.917000] SCSI subsystem initialized [ 0.918071] ACPI: bus type USB registered [ 0.919093] usbcore: registered new interface driver usbfs [ 0.920062] usbcore: registered new interface driver hub [ 0.922060] usbcore: registered new device driver usb [ 0.923153] pps_core: LinuxPPS API ver. 1 registered [ 0.924009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.926057] PTP clock support registered [ 0.928062] EDAC MC: Ver: 3.0.0 [ 0.929259] PCI: Using ACPI for IRQ routing [ 0.930571] NetLabel: Initializing [ 0.932013] NetLabel: domain hash size = 128 [ 0.933015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.935087] NetLabel: unlabeled traffic allowed by default [ 0.939043] vgaarb: loaded [ 0.940252] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.942017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.949162] clocksource: Switched to clocksource kvm-clock [ 1.050512] VFS: Disk quotas dquot_6.6.0 [ 1.054198] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.057577] *** VALIDATE ramfs *** [ 1.059111] *** VALIDATE hugetlbfs *** [ 1.060953] pnp: PnP ACPI init [ 1.063382] pnp: PnP ACPI: found 6 devices [ 1.087135] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.091085] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.093285] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.096156] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.098787] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.101341] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.104416] NET: Registered protocol family 2 [ 1.106924] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.112068] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.115499] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.121127] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.124482] TCP: Hash tables configured (established 65536 bind 65536) [ 1.127446] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.130536] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.133429] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.137157] NET: Registered protocol family 1 [ 1.139570] RPC: Registered named UNIX socket transport module. [ 1.141705] RPC: Registered udp transport module. [ 1.143607] RPC: Registered tcp transport module. [ 1.145348] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.147914] NET: Registered protocol family 44 [ 1.149684] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.151516] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.153726] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.156062] PCI: CLS 0 bytes, default 64 [ 1.157537] Unpacking initramfs... [ 2.547961] debug: unmapping init [mem 0xffffa0db3cc54000-0xffffa0db3ffbffff] [ 2.551752] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.554202] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.557098] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.080620] Initialise system trusted keyrings [ 3.082135] Key type blacklist registered [ 3.083751] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.092404] zbud: loaded [ 3.095226] *** VALIDATE nfs *** [ 3.096250] *** VALIDATE nfs4 *** [ 3.097634] pstore: using deflate compression [ 3.100786] Platform Keyring initialized [ 3.202916] NET: Registered protocol family 38 [ 3.204501] Key type asymmetric registered [ 3.205782] Asymmetric key parser 'x509' registered [ 3.207709] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.211071] io scheduler mq-deadline registered [ 3.212448] io scheduler kyber registered [ 3.213843] io scheduler bfq registered [ 3.216153] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.219138] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.221995] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.224910] ACPI: Power Button [PWRF] [ 3.230428] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.238600] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.249545] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.256520] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.269846] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.298145] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.328685] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.333542] Non-volatile memory driver v1.3 [ 3.335287] Linux agpgart interface v0.103 [ 3.365572] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.367679] vda: detected capacity change from 0 to 74702848 [ 3.380650] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.383685] vdb: detected capacity change from 0 to 1073741824 [ 3.398570] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.401532] vdc: detected capacity change from 0 to 2621440000 [ 3.419570] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.422727] vdd: detected capacity change from 0 to 2621440000 [ 3.438711] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.442281] vde: detected capacity change from 0 to 4294967296 [ 3.457948] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.461368] vdf: detected capacity change from 0 to 4294967296 [ 3.468584] libphy: Fixed MDIO Bus: probed [ 3.481453] usbcore: registered new interface driver usbserial_generic [ 3.485016] usbserial: USB Serial support registered for generic [ 3.487641] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.492356] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.494394] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.498363] mousedev: PS/2 mouse device common for all mice [ 3.500941] rtc_cmos 00:05: RTC can wake from S4 [ 3.504133] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.508085] rtc_cmos 00:05: registered as rtc0 [ 3.510624] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.511023] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.515557] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.517692] intel_pstate: CPU model not supported [ 3.524089] hid: raw HID events driver (C) Jiri Kosina [ 3.525986] usbcore: registered new interface driver usbhid [ 3.528105] usbhid: USB HID core driver [ 3.529680] drop_monitor: Initializing network drop monitor service [ 3.532034] Initializing XFRM netlink socket [ 3.534080] NET: Registered protocol family 10 [ 3.536664] Segment Routing with IPv6 [ 3.538063] NET: Registered protocol family 17 [ 3.539932] mpls_gso: MPLS GSO support [ 3.546530] RAS: Correctable Errors collector initialized. [ 3.548692] AVX version of gcm_enc/dec engaged. [ 3.550599] AES CTR mode by8 optimization enabled [ 3.629630] sched_clock: Marking stable (3629600062, 0)->(4562216921, -932616859) [ 3.633050] registered taskstats version 1 [ 3.634720] Loading compiled-in X.509 certificates [ 3.636907] zswap: loaded using pool lzo/zbud [ 3.662673] Key type big_key registered [ 3.675622] Key type encrypted registered [ 3.677129] ima: No TPM chip found, activating TPM-bypass! [ 3.679022] ima: Allocated hash algorithm: sha1 [ 3.680513] ima: No architecture policies found [ 3.682195] evm: Initialising EVM extended attributes: [ 3.683755] evm: security.selinux [ 3.684956] evm: security.ima [ 3.686028] evm: security.capability [ 3.687265] evm: HMAC attrs: 0x1 [ 3.689264] rtc_cmos 00:05: setting system clock to 2026-08-15 16:48:17 UTC (1786812497) [ 3.695115] debug: unmapping init [mem 0xffffffff9ea03000-0xffffffff9ebfffff] [ 3.697771] debug: unmapping init [mem 0xffffffff9d782000-0xffffffff9da58fff] [ 3.710072] Write protecting the kernel read-only data: 28672k [ 3.713177] debug: unmapping init [mem 0xffffffff9be03000-0xffffffff9bffffff] [ 3.715757] debug: unmapping init [mem 0xffffffff9c714000-0xffffffff9c7fffff] [ 3.750370] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.758099] systemd[1]: Detected virtualization kvm. [ 3.759665] systemd[1]: Detected architecture x86-64. [ 3.761552] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.785691] systemd[1]: No hostname configured. [ 3.787328] systemd[1]: Set hostname to . [ 3.789349] random: systemd: uninitialized urandom read (16 bytes read) [ 3.791847] systemd[1]: Initializing machine ID from random generator. [ 3.836189] random: ln: uninitialized urandom read (6 bytes read) [ 3.917215] random: systemd: uninitialized urandom read (16 bytes read) [ 3.919559] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.923973] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.927578] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.500376] device-mapper: uevent: version 1.0.3 [ 4.503213] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.228293] random: fast init done [ 5.240916] virtio_net virtio0 ens2: renamed from eth0 [ 5.314433] scsi host0: ata_piix [ 5.322176] scsi host1: ata_piix [ 5.323816] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.326290] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.179769] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.191636] random: crng init done [ 10.193742] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.616178] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.702233] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.955761] SELinux: Disabled at runtime. [ 12.020417] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.032429] systemd[1]: Detected virtualization kvm. [ 12.035247] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.506893] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.511055] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.517667] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.520674] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.524090] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.530389] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.533871] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Local Encrypted Volumes. [ 12.694846] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.940237] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.234264] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.299738] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.396462] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.406369] EDAC sbridge: Ver: 1.1.2 [ 14.960564] Key type dns_resolver registered [ 15.242837] NFS: Registering the id_resolver key type [ 15.245577] Key type id_resolver registered [ 15.248044] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg251-server login: [ 41.846059] hrtimer: interrupt took 4013957 ns [ 82.882675] libcfs: loading out-of-tree module taints kernel. [ 82.923910] Key type ._llcrypt registered [ 82.926621] Key type .llcrypt registered [ 83.018843] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 104.084462] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 106.551351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 106.569411] alg: No test for adler32 (adler32-zlib) [ 107.963251] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 109.074833] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 110.863196] Key type lgssc registered [ 112.710359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 134.307821] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 179.848891] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 194.980544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 195.022495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.499690] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 196.579550] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 196.776114] Lustre: lustre-MDT0000: new disk, initializing [ 196.891925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.907937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 201.761531] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.867570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.980317] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 217.012259] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 217.017635] Lustre: Skipped 1 previous similar message [ 217.109667] Lustre: lustre-MDT0001: new disk, initializing [ 217.193533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 217.233319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 217.251204] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 222.387738] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 227.561048] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 238.675826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 239.007617] Lustre: lustre-OST0000: new disk, initializing [ 239.010697] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 239.017228] Lustre: 8451:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 239.106051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 242.683664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 242.708825] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 242.873567] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 247.451580] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 265.323670] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 265.573941] Lustre: lustre-OST0001: new disk, initializing [ 265.582428] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 265.598147] Lustre: 9526:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 265.670623] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 271.959276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 271.972772] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 272.120138] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 273.946627] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 286.907845] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 296.643846] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 303.611789] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing check_logdir /tmp/testlogs/ [ 309.218784] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing yml_node [ 314.601745] Lustre: DEBUG MARKER: Client: 2.17.57.2 [ 317.545169] Lustre: DEBUG MARKER: MDS: 2.17.57.2 [ 320.766972] Lustre: DEBUG MARKER: OSS: 2.17.57.2 [ 322.837827] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 15 12:53:33 EDT 2026 [ 343.488214] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 345.346590] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 364.513478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.515300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 364.532596] Lustre: Skipped 3 previous similar messages [ 364.556358] Lustre: Skipped 3 previous similar messages [ 368.060047] Lustre: server umount lustre-MDT0000 complete [ 372.589518] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786812866 with bad export cookie 6668122064680802223 [ 372.600857] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 372.603190] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 374.752874] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 374.757305] LustreError: 6520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 374.762241] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 374.768690] Lustre: Skipped 1 previous similar message [ 374.794443] LustreError: 6520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 377.761122] LustreError: 6521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 377.763139] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 377.788272] LustreError: 6521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 377.808099] Lustre: Skipped 2 previous similar messages [ 379.129969] Lustre: server umount lustre-MDT0001 complete [ 383.700477] Lustre: server umount lustre-OST0000 complete [ 388.386640] Lustre: server umount lustre-OST0001 complete [ 397.356919] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 410.146699] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 455.633640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 455.935186] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 455.964790] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 456.063675] Lustre: lustre-MDT0000: new disk, initializing [ 456.151779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 456.169751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 461.813465] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 476.230236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 476.416558] Lustre: 15080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 476.558749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 476.564427] Lustre: Skipped 1 previous similar message [ 476.725844] Lustre: lustre-MDT0001: new disk, initializing [ 476.846324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 476.886976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 476.897910] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 482.535406] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 494.244129] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 502.547763] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 510.432407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 510.820988] Lustre: lustre-OST0000: new disk, initializing [ 510.825615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 510.838666] Lustre: 17051:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 510.980150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 512.249122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 512.263377] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 512.494030] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 519.194909] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 536.010475] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 543.131942] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 543.465685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 549.936155] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 550.362402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 553.449954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.452646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 553.481211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 553.493427] Lustre: Skipped 1 previous similar message [ 558.578395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 558.918941] Lustre: server umount lustre-OST0000 complete [ 563.687448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 563.702709] Lustre: Skipped 2 previous similar messages [ 568.805809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 568.821176] Lustre: Skipped 4 previous similar messages [ 569.549712] Lustre: server umount lustre-MDT0000 complete [ 573.222351] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813067 with bad export cookie 6668122064680803756 [ 573.223309] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 573.232989] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 573.663553] Lustre: server umount lustre-MDT0001 complete [ 580.181166] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 12:57:51 (1786813071) [ 605.508567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 606.246574] LustreError: 19929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 606.379060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 611.810259] LustreError: 19930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 612.278458] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 616.928678] LustreError: 19929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 622.154834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 627.860904] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 636.400659] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 643.139480] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 645.607218] Lustre: Failing over lustre-MDT0000 [ 645.865841] Lustre: server umount lustre-MDT0000 complete [ 647.649073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 647.662778] LustreError: 19930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 647.694087] LustreError: 19930:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 663.903463] Lustre: 3646:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786813141/real 1786813141] req@ffffa0db8fc23b80 x1873608811125120/t0(0) o400->MGC192.168.202.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786813157 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 663.930031] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 667.297143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 668.066108] LustreError: 19930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 668.087376] LustreError: 19930:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 674.645370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 674.656875] Lustre: Skipped 1 previous similar message [ 674.726210] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 679.765303] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 679.928383] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 679.932813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 679.956470] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 691.011805] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 695.268993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 695.279505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 695.301896] Lustre: Skipped 1 previous similar message [ 695.325413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 701.439715] Lustre: server umount lustre-MDT0000 complete [ 702.451587] LustreError: 19929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 702.486964] LustreError: 19929:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 704.971362] LustreError: 19911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813198 with bad export cookie 6668122064680805170 [ 704.983814] LustreError: 19911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 704.988169] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 705.011986] LustreError: 3646:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [ 705.397993] Lustre: server umount lustre-MDT0001 complete [ 716.571714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 724.729178] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 733.518105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 739.884429] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 748.170896] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 755.097508] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 757.102377] Lustre: Failing over lustre-MDT0000 [ 757.493470] Lustre: server umount lustre-MDT0000 complete [ 758.767232] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 758.801391] Lustre: Skipped 1 previous similar message [ 770.018596] LustreError: 23271:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 770.042978] LustreError: 23271:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 775.090230] Lustre: 3648:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786813252/real 1786813252] req@ffffa0da8379d180 x1873608811155584/t0(0) o400->MGC192.168.202.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786813268 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 775.128807] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 778.321046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 784.355693] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x5c89edb0483e66dc [ 784.371728] Lustre: MGC192.168.202.151@tcp: Connection restored to 0@lo (at 0@lo) [ 784.398742] Lustre: Skipped 1 previous similar message [ 784.854617] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 789.992706] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 790.005206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 790.042446] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 792.054741] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 804.669987] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 810.475892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 810.497452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 810.507665] Lustre: Skipped 1 previous similar message [ 810.522215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 810.533606] Lustre: Skipped 2 previous similar messages [ 824.287923] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 824.631570] Lustre: server umount lustre-MDT0000 complete [ 829.388321] LustreError: 23252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813323 with bad export cookie 6668122064680806108 [ 829.392432] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 829.398655] LustreError: 23252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 829.712882] Lustre: server umount lustre-MDT0001 complete [ 840.904872] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 13:02:11 (1786813331) [ 861.805454] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 876.858593] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 931.183176] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 944.136051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 944.376545] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 944.397215] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 944.455898] Lustre: lustre-MDT0000: new disk, initializing [ 944.524396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 944.529318] Lustre: Skipped 3 previous similar messages [ 944.546494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 949.307474] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 962.450773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 962.622343] Lustre: 29688:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 962.943926] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 968.302518] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 979.560219] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 985.956887] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 993.805715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 994.134360] Lustre: 31658:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 995.188815] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 995.216026] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 995.253195] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 995.260680] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 999.604273] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1011.400293] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1018.628438] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1041.380700] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1041.394611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1041.405863] Lustre: Skipped 2 previous similar messages [ 1041.410479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1041.413659] Lustre: Skipped 5 previous similar messages [ 1055.199175] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1055.537629] Lustre: server umount lustre-OST0000 complete [ 1066.987711] LustreError: 30532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1067.014197] LustreError: 30532:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 1067.238433] Lustre: server umount lustre-MDT0000 complete [ 1070.901966] LustreError: 29680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813564 with bad export cookie 6668122064680806521 [ 1070.909729] LustreError: 29680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1070.913434] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1071.532638] Lustre: server umount lustre-MDT0001 complete [ 1085.422309] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 1088.174509] Key type lgssc unregistered [ 1088.498199] LNet: 33640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1088.503134] LNetError: 33640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1088.519926] LNet: Removed LNI 192.168.202.151@tcp [ 1089.734168] Key type .llcrypt unregistered [ 1089.735164] Key type ._llcrypt unregistered [ 1116.970194] Key type ._llcrypt registered [ 1116.975331] Key type .llcrypt registered [ 1117.168857] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 1118.468908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1118.707197] alg: No test for adler32 (adler32-zlib) [ 1119.764282] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 1120.045607] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 1121.727192] Key type lgssc registered [ 1123.108440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1132.268403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1132.318827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1134.303418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1140.910804] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1143.785917] LustreError: 35056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1143.825636] LustreError: 35056:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1148.908848] LustreError: 35057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1149.933154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1150.381823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1155.717449] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1165.111641] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1171.492764] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1179.570668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1180.079125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1186.536778] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1194.911597] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1195.446342] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1203.673475] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1219.661952] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 13:08:30 (1786813710) [ 1222.033303] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1223.986740] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1231.302303] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 13:08:42 (1786813722) [ 1240.343575] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 13:08:51 (1786813731) [ 1241.487068] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1249.669094] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1249.678646] Lustre: Skipped 2 previous similar messages [ 1252.880757] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 13:09:02 (1786813742) [ 1255.125495] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1255.129614] Lustre: Skipped 2 previous similar messages [ 1266.520225] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1266.539605] Lustre: Skipped 4 previous similar messages [ 1283.827875] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1283.833632] Lustre: Skipped 2 previous similar messages [ 1287.152405] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 13:09:37 (1786813777) [ 1290.312448] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1294.343277] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1304.029703] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 13:09:55 (1786813795) [ 1318.749432] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 1318.754745] Lustre: Skipped 7 previous similar messages [ 1320.541325] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 13:10:11 (1786813811) [ 1336.802281] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1336.804386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1336.807353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1336.835803] Lustre: Skipped 1 previous similar message [ 1341.269352] Lustre: server umount lustre-OST0000 complete [ 1347.042248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1347.043845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1347.058448] Lustre: Skipped 1 previous similar message [ 1347.066629] Lustre: Skipped 2 previous similar messages [ 1352.162973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1352.172402] Lustre: Skipped 1 previous similar message [ 1356.259868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1356.263902] Lustre: Skipped 1 previous similar message [ 1359.327302] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1359.489354] Lustre: server umount lustre-MDT0000 complete [ 1361.382773] LustreError: 35057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1361.417430] LustreError: 35057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1363.473884] LustreError: 35037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813857 with bad export cookie 10213751477157536493 [ 1363.477807] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1363.488322] LustreError: 35037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1363.863907] Lustre: server umount lustre-MDT0001 complete [ 1374.630761] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 1377.208599] Key type lgssc unregistered [ 1377.552049] LNet: 42938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1377.557578] LNetError: 42938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1377.580276] LNet: Removed LNI 192.168.202.151@tcp [ 1378.391258] Key type .llcrypt unregistered [ 1378.393301] Key type ._llcrypt unregistered [ 1394.720076] Key type ._llcrypt registered [ 1394.721327] Key type .llcrypt registered [ 1394.839813] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 1413.709660] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 1414.818617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1414.852522] alg: No test for adler32 (adler32-zlib) [ 1415.933646] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 1416.164293] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 1417.855490] Key type lgssc registered [ 1418.956882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1460.011937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1460.041649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1461.324850] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1461.343160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1461.402623] Lustre: lustre-MDT0000: new disk, initializing [ 1461.509393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1461.548826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1465.543770] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1476.245034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1476.403966] Lustre: 46094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1476.454161] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1476.457241] Lustre: Skipped 1 previous similar message [ 1476.512352] Lustre: lustre-MDT0001: new disk, initializing [ 1476.570825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1476.622668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1476.630329] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1480.411459] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1488.826540] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1493.881565] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1500.637298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1500.972202] Lustre: lustre-OST0000: new disk, initializing [ 1500.974881] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1500.979962] Lustre: 48058:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1501.071115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1507.317845] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1508.419077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1508.428462] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1508.554189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1516.918413] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1522.359312] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1522.590487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1527.590631] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1527.819764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1534.433181] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1534.433860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1534.451227] LustreError: Skipped 1 previous similar message [ 1534.456347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1534.481830] Lustre: Skipped 1 previous similar message [ 1534.518258] Lustre: Skipped 1 previous similar message [ 1535.666492] Lustre: server umount lustre-OST0000 complete [ 1539.553230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1539.556157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1539.580629] Lustre: Skipped 1 previous similar message [ 1544.674947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1544.682027] Lustre: Skipped 2 previous similar messages [ 1545.363621] Lustre: server umount lustre-MDT0000 complete [ 1549.343367] LustreError: 46085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814043 with bad export cookie 17819054000201733699 [ 1549.362043] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1549.675510] Lustre: server umount lustre-MDT0001 complete [ 1560.607305] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 13:14:11 (1786814051) [ 1569.207389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1569.908457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1574.417437] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1575.393929] LustreError: 50519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1575.407687] LustreError: 50519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1579.499950] LustreError: 50518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1583.689874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1584.288933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1588.988402] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1595.886985] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1601.835777] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1608.484561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1608.760596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1614.671994] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1623.373947] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1629.017419] LustreError: 53045:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1629.992199] Lustre: Modifying parameter general.timeout=1 in log params [ 1631.372540] Lustre: Modifying parameter general.timeout=2 in log params [ 1633.675763] Lustre: Modifying parameter general.timeout=4 in log params [ 1633.683330] Lustre: Skipped 1 previous similar message [ 1638.090590] Lustre: Modifying parameter general.timeout=8 in log params [ 1638.093336] Lustre: Skipped 3 previous similar messages [ 1646.661163] Lustre: Modifying parameter general.timeout=18 in log params [ 1646.663422] Lustre: Skipped 9 previous similar messages [ 1663.489028] Lustre: Modifying parameter general.timeout=36 in log params [ 1663.495735] Lustre: Skipped 17 previous similar messages [ 1695.759735] Lustre: Modifying parameter general.timeout=68 in log params [ 1695.774224] Lustre: Skipped 31 previous similar messages [ 1760.891951] Lustre: Disabling parameter general.timeout= in log params [ 1760.894195] Lustre: Skipped 61 previous similar messages [ 1764.415848] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 13:17:35 (1786814255) [ 1767.482897] Lustre: 60002:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1769.836723] Lustre: 60100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1773.790150] Lustre: 60306:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1773.803985] Lustre: 60306:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1804.216275] Lustre: 61863:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1804.224243] Lustre: 61863:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1804.249340] LustreError: 61863:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1808.480929] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 13:18:19 (1786814299) [ 1823.204204] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1823.210123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1823.233069] Lustre: Skipped 1 previous similar message [ 1826.945915] Lustre: server umount lustre-MDT0000 complete [ 1828.321658] LustreError: 50514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1828.356629] LustreError: 50514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1830.794946] LustreError: 54582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814324 with bad export cookie 17819054000201734581 [ 1830.815457] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1830.823406] LustreError: 54582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1831.278205] Lustre: server umount lustre-MDT0001 complete [ 1845.818710] Lustre: server umount lustre-OST0000 complete [ 1883.804310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1884.043169] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1884.408362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1890.323259] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1900.982237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1901.065678] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1901.090789] Lustre: 64397:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1906.736375] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1915.425569] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1921.397127] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1929.024229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1929.244554] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1929.583897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1929.594803] Lustre: Skipped 1 previous similar message [ 1936.049494] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1943.559597] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1945.010485] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1945.021724] Lustre: Skipped 23 previous similar messages [ 1945.070515] Lustre: 66965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1956.320771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1956.326503] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1956.334598] Lustre: Skipped 1 previous similar message [ 1956.340130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1956.348086] Lustre: Skipped 2 previous similar messages [ 1958.263451] Lustre: server umount lustre-OST0000 complete [ 1961.954455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1961.964719] Lustre: Skipped 2 previous similar messages [ 1967.072731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1967.081982] Lustre: Skipped 4 previous similar messages [ 1976.290090] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1976.547372] Lustre: server umount lustre-MDT0000 complete [ 1980.160717] LustreError: 64390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814473 with bad export cookie 17819054000201737640 [ 1980.170240] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1980.173189] LustreError: 64390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1980.467288] Lustre: server umount lustre-MDT0001 complete [ 1990.526425] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 1993.382350] Key type lgssc unregistered [ 1993.804470] LNet: 68175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1993.817418] LNetError: 68175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1993.831483] LNet: Removed LNI 192.168.202.151@tcp [ 1994.737719] Key type .llcrypt unregistered [ 1994.739082] Key type ._llcrypt unregistered [ 2017.683415] Key type ._llcrypt registered [ 2017.685396] Key type .llcrypt registered [ 2017.777403] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 2018.559889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2018.638365] alg: No test for adler32 (adler32-zlib) [ 2019.735609] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 2019.953812] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 2021.607373] Key type lgssc registered [ 2022.627713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2030.893651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2030.913594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2032.686102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2036.836941] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2042.861879] LustreError: 69588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2042.882589] LustreError: 69588:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2046.690420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2047.501147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2051.839444] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2061.002881] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2067.640640] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2075.640615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2076.053238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2083.169371] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2091.478390] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2099.613480] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2111.969310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2111.982947] Lustre: Skipped 1 previous similar message [ 2111.984781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2111.997258] Lustre: Skipped 2 previous similar messages [ 2116.072785] Lustre: server umount lustre-MDT0000 complete [ 2117.094894] LustreError: 69589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2117.108392] LustreError: 69589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2120.118906] LustreError: 69571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814613 with bad export cookie 2396459373894624227 [ 2120.146249] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2120.636739] Lustre: server umount lustre-MDT0001 complete [ 2135.282792] Lustre: server umount lustre-OST0000 complete [ 2163.024494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2163.310508] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2163.346713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2163.524960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2168.324241] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2178.044039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2178.141867] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2178.161660] Lustre: 74463:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2178.887219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2185.379492] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2196.929497] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2204.633451] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2212.549704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2212.948766] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2213.580343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2220.535734] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2229.772732] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2232.398491] Lustre: 77038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2235.122782] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2242.532230] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2242.537375] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2242.570312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2247.136911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2247.141730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2247.157368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2248.642369] Lustre: server umount lustre-OST0000 complete [ 2252.772342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2252.776581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2252.782927] Lustre: Skipped 1 previous similar message [ 2252.791735] Lustre: Skipped 1 previous similar message [ 2257.891912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2257.903305] Lustre: Skipped 1 previous similar message [ 2258.655760] Lustre: server umount lustre-MDT0000 complete [ 2262.562292] LustreError: 74453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814756 with bad export cookie 2396459373894625074 [ 2262.573848] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2262.576503] LustreError: 74453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2263.362731] Lustre: server umount lustre-MDT0001 complete [ 2276.042628] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 2280.144436] Key type lgssc unregistered [ 2280.582622] LNet: 78254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2280.599427] LNetError: 78254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2280.630674] LNet: Removed LNI 192.168.202.151@tcp [ 2281.535422] Key type .llcrypt unregistered [ 2281.538383] Key type ._llcrypt unregistered [ 2309.703794] Key type ._llcrypt registered [ 2309.705750] Key type .llcrypt registered [ 2309.871835] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 2310.571942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2310.675829] alg: No test for adler32 (adler32-zlib) [ 2311.761521] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 2312.198154] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 2313.911158] Key type lgssc registered [ 2315.522685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2324.553898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2324.596647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2326.418062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2332.264911] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2336.745233] LustreError: 79666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2336.757783] LustreError: 79666:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2341.863160] LustreError: 79665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2344.090879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2345.267072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2351.264808] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2360.046202] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2366.063754] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2372.982855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2373.400078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2380.290708] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2389.041477] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2406.963614] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2407.953202] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2409.322572] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2412.396135] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2412.401661] Lustre: Skipped 2 previous similar messages [ 2416.883086] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2416.885996] Lustre: Skipped 3 previous similar messages [ 2425.302949] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2425.308647] Lustre: Skipped 6 previous similar messages [ 2441.881764] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2441.884447] Lustre: Skipped 14 previous similar messages [ 2474.671348] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2474.675874] Lustre: Skipped 34 previous similar messages [ 2517.946007] Lustre: 87797:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2521.040100] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 13:30:12 (1786815012) [ 2522.587476] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2524.486920] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 13:30:15 (1786815015) [ 2530.277898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2530.286149] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2530.300313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2530.319074] Lustre: Skipped 1 previous similar message [ 2534.887927] Lustre: server umount lustre-OST0000 complete [ 2539.489108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2539.494360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2539.510889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2539.517124] Lustre: Skipped 1 previous similar message [ 2540.512662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2540.516836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2544.597643] LustreError: 88294:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (0000000009fc4425) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 2545.632805] LustreError: 79665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2545.651435] LustreError: 79665:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2546.729627] Lustre: server umount lustre-MDT0000 complete [ 2550.768777] LustreError: 79661:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2550.779206] LustreError: 79646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815044 with bad export cookie 2186913782463193092 [ 2550.786873] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2550.831969] LustreError: 79661:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2551.190792] Lustre: server umount lustre-MDT0001 complete [ 2562.209759] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 2565.780686] Key type lgssc unregistered [ 2566.289826] LNet: 89049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2566.295752] LNetError: 89049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2566.311926] LNet: Removed LNI 192.168.202.151@tcp [ 2567.429243] Key type .llcrypt unregistered [ 2567.436142] Key type ._llcrypt unregistered [ 2572.384749] Key type ._llcrypt registered [ 2572.392782] Key type .llcrypt registered [ 2572.574255] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2579.978750] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules [ 2581.732922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2581.748160] alg: No test for adler32 (adler32-zlib) [ 2582.872980] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 2583.147807] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 2584.827381] Key type lgssc registered [ 2586.743715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2588.315338] LustreError: 90309:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2610.071379] LustreError: 90309:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2614.984909] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 2624.623796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2624.665152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2626.287945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2630.804873] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2635.753066] LustreError: 91332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2635.779988] LustreError: 91332:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2639.578265] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 13:32:10 (1786815130) [ 2640.879321] LustreError: 91331:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2644.924777] LustreError: 91363:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2644.936896] LustreError: 91363:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2644.948987] LustreError: 91363:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2645.462589] Lustre: server umount lustre-MDT0000 complete [ 2663.332668] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 2666.344118] Key type lgssc unregistered [ 2666.765738] LNet: 92859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2666.774675] LNetError: 92859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2666.800656] LNet: Removed LNI 192.168.202.151@tcp [ 2668.077608] Key type .llcrypt unregistered [ 2668.082347] Key type ._llcrypt unregistered [ 2690.431677] Key type ._llcrypt registered [ 2690.433310] Key type .llcrypt registered [ 2690.560657] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 2691.413119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2691.489814] alg: No test for adler32 (adler32-zlib) [ 2692.564804] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 2692.756568] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 2694.599417] Key type lgssc registered [ 2696.123222] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2711.460807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2711.516154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2713.421228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2720.578959] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2723.808906] LustreError: 94079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2723.833276] LustreError: 94079:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2728.942800] LustreError: 94080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2730.083938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2731.288796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2736.462925] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2746.080924] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2754.445193] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2765.451372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2766.167350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2773.708143] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2781.725406] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2833.450566] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 13:35:24 (1786815324) [ 2835.291377] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2837.897977] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 13:35:28 (1786815328) [ 2843.622389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2843.632719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2843.654516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2845.152474] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2845.154844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2845.178431] Lustre: Skipped 2 previous similar messages [ 2849.440660] Lustre: server umount lustre-MDT0000 complete [ 2850.278681] LustreError: 94075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2850.301122] LustreError: 94075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2853.324122] LustreError: 94061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815347 with bad export cookie 9457286688625184865 [ 2853.345052] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2853.996386] Lustre: server umount lustre-MDT0001 complete [ 2869.017940] Lustre: server umount lustre-OST0000 complete [ 2882.808966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2883.335256] LustreError: 98357:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2883.664721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2888.684247] LustreError: 98358:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2889.884337] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2898.930129] LustreError: 98358:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2898.946851] LustreError: 98358:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2900.388371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2901.360719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2906.768081] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2915.249663] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2921.380648] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2938.411384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2938.621630] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2938.626594] LustreError: 98345:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2938.645769] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2938.650842] LustreError: 100443:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2938.655882] LustreError: 100443:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2938.660287] LustreError: 100443:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2938.710289] Lustre: server umount lustre-OST0000 complete [ 2938.712733] LustreError: 100443:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2949.530549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2949.789576] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2949.800623] LustreError: 98345:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2949.838544] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2949.850212] LustreError: 100910:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2949.858510] LustreError: 100910:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2949.866211] LustreError: 100910:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2949.977587] Lustre: server umount lustre-OST0000 complete [ 2949.980921] LustreError: 100910:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2960.563635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2960.715252] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2960.719865] Lustre: Found index 0 for lustre-OST0000, updating log [ 2960.727446] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2960.812433] Lustre: lustre-OST0000: new disk, initializing [ 2960.820329] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2960.824859] Lustre: 101445:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2961.089971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2968.393578] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2972.688257] Lustre: Failing over lustre-OST0000 [ 2972.868651] Lustre: server umount lustre-OST0000 complete [ 2977.762376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2977.777909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2977.789182] Lustre: Skipped 1 previous similar message [ 2977.798332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2980.832095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2980.849950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2983.279042] Lustre: server umount lustre-MDT0000 complete [ 2985.958225] LustreError: 98352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2985.993536] LustreError: 98352:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2987.839293] LustreError: 98337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815481 with bad export cookie 9457286688625187609 [ 2987.843958] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2987.849953] LustreError: 98337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2997.455353] Lustre: server umount lustre-MDT0001 complete [ 3008.652524] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 13:38:19 (1786815499) [ 3028.110464] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3040.085776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3040.467298] LustreError: 103795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3040.633365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3045.115371] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3055.554715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3062.925879] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3066.731176] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3075.579450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3076.517150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3076.522777] Lustre: Skipped 1 previous similar message [ 3076.548850] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3080.616545] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3080.658698] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3080.661881] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3084.563502] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3095.277765] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3095.391378] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3095.502408] Lustre: lustre-OST0001: new disk, initializing [ 3095.507540] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3095.514970] Lustre: 106113:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3103.541419] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3104.400215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3104.420564] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3104.511137] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3112.977730] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3123.307868] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3128.290833] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3128.300285] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3128.317261] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3128.324454] Lustre: Skipped 1 previous similar message [ 3134.184291] Lustre: server umount lustre-MDT0001 complete [ 3135.470031] LustreError: 103790:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3135.542893] LustreError: 103790:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3144.272608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3144.469879] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3144.476123] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3144.569271] Lustre: 103785:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3145.721484] LustreError: 107619:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3145.744856] LustreError: 107619:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3145.777556] LustreError: 103787:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3145.883409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3145.890476] Lustre: Skipped 1 previous similar message [ 3146.016186] LustreError: 107634:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.151@tcp: cfg command failed: rc = -17 [ 3146.045205] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3146.045205] [ 3151.353151] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3152.320869] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3166.274761] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 13:40:56 (1786815656) [ 3187.868227] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3209.920532] Lustre: 109728:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3237.911098] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3242.912498] Lustre: 110864:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3339.746061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3339.761309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3339.777760] Lustre: Skipped 2 previous similar messages [ 3339.788357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3339.806719] Lustre: Skipped 4 previous similar messages [ 3344.979612] Lustre: server umount lustre-MDT0000 complete [ 3345.888565] LustreError: 103792:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3345.916417] LustreError: 103792:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 3350.114357] LustreError: 103776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815843 with bad export cookie 9457286688625188610 [ 3350.135460] LustreError: 103776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3350.139337] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3351.021489] Lustre: server umount lustre-MDT0001 complete [ 3365.910937] Lustre: server umount lustre-OST0000 complete [ 3367.199143] Lustre: 93389:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786815844/real 1786815844] req@ffffa0dba7f5d180 x1873611520861440/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786815860 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3367.225343] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3367.246405] Lustre: Skipped 3 previous similar messages [ 3370.464749] Lustre: 93392:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786815848/real 1786815848] req@ffffa0dbb4a55500 x1873611520861696/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786815864 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3370.738326] Lustre: server umount lustre-OST0001 complete [ 3378.072602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3391.412818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3401.519943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3412.537691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3436.224141] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3449.032452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3449.050211] Lustre: lustre-MDT0000: reset Object Index mappings [ 3449.724723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3455.201905] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3464.976818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3465.008046] Lustre: lustre-MDT0001: reset Object Index mappings [ 3471.777826] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3475.424884] Lustre: 115970:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3483.964373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3491.234287] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3492.088762] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3495.362904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3503.184605] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3509.769580] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3509.784049] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3511.950651] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3521.466174] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3526.247690] Lustre: 117847:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3586.197246] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 13:47:57 (1786816077) [ 3612.158597] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3612.161307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3612.191635] Lustre: Skipped 3 previous similar messages [ 3612.225086] Lustre: Skipped 5 previous similar messages [ 3618.697066] Lustre: server umount lustre-MDT0000 complete [ 3622.374460] LustreError: 114818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3622.405586] LustreError: 114818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 3623.893885] LustreError: 114802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816117 with bad export cookie 9457286688625235468 [ 3623.920693] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3623.922886] LustreError: 114802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3630.506908] Lustre: server umount lustre-MDT0001 complete [ 3642.129522] Lustre: server umount lustre-OST0000 complete [ 3653.134632] Lustre: server umount lustre-OST0001 complete [ 3661.136422] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 3672.741794] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3722.307906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3722.566832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3722.630269] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3722.744291] Lustre: lustre-MDT0000: new disk, initializing [ 3722.866862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3722.877154] Lustre: Skipped 3 previous similar messages [ 3722.900680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3728.413578] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3738.149648] Lustre: server umount lustre-MDT0000 complete [ 3749.367379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3749.671834] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3749.685809] Lustre: 122132:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3749.697299] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3756.238541] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3762.741074] Lustre: server umount lustre-MDT0000 complete [ 3772.596898] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 13:51:03 (1786816263) [ 3774.131517] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3776.241428] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 13:51:07 (1786816267) [ 3796.844179] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3808.312757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3813.469936] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3823.030719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3823.126387] Lustre: 124181:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3823.170694] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3823.173695] Lustre: Skipped 1 previous similar message [ 3823.225598] Lustre: lustre-MDT0001: new disk, initializing [ 3823.388129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3823.397186] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3828.544692] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3833.632966] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3840.941365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3841.314938] Lustre: lustre-OST0000: new disk, initializing [ 3841.322715] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3841.332689] Lustre: 125759:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3843.357442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3843.368354] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3843.549605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3848.494599] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3860.020768] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3860.261378] Lustre: lustre-OST0001: new disk, initializing [ 3860.263893] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3860.269413] Lustre: 126630:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3861.659530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3861.672902] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3861.966690] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3867.749682] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3878.699669] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3883.270579] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4358.083705] Lustre: 133064:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4358.105571] Lustre: 133064:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4358.124358] Lustre: 133064:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4358.142848] Lustre: 133064:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4358.166483] Lustre: 133064:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4358.185721] Lustre: 133064:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4659.871063] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:05:49 (1786817149) [ 4678.628060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4678.647498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4678.670363] Lustre: Skipped 2 previous similar messages [ 4678.683600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4678.701332] Lustre: Skipped 10 previous similar messages [ 4683.041138] Lustre: server umount lustre-MDT0000 complete [ 4685.794443] LustreError: 124187:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4685.817487] LustreError: 124187:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4687.035900] LustreError: 124173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817180 with bad export cookie 9457286688625305328 [ 4687.039580] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4687.061880] LustreError: 124173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4690.936085] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4690.950837] Lustre: Skipped 3 previous similar messages [ 4690.957962] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4690.969874] Lustre: Skipped 5 previous similar messages [ 4693.776819] Lustre: server umount lustre-MDT0001 complete [ 4703.796367] Lustre: server umount lustre-OST0000 complete [ 4714.485691] Lustre: server umount lustre-OST0001 complete [ 4722.433782] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 4736.704777] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 4780.510790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4780.802745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4780.837588] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4780.929652] Lustre: lustre-MDT0000: new disk, initializing [ 4781.038038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4781.052836] Lustre: Skipped 5 previous similar messages [ 4781.085234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4787.414726] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4800.260326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4800.403290] Lustre: 136132:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4800.452680] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4800.456126] Lustre: Skipped 1 previous similar message [ 4800.514652] Lustre: lustre-MDT0001: new disk, initializing [ 4800.640595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4800.666027] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4806.272520] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4816.218833] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4822.090620] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4829.241438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4829.532303] Lustre: lustre-OST0000: new disk, initializing [ 4829.534703] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4829.540626] Lustre: 138102:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4831.524234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4831.538803] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4831.734408] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4838.268919] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4850.521288] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4859.723497] Lustre: lustre-MDD0000: changelog on [ 4862.548520] Lustre: lustre-MDD0001: changelog on [ 6665.725547] Lustre: lustre-MDD0001: changelog off [ 6667.548441] Lustre: 137423:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6668.489420] Lustre: lustre-MDD0000: changelog off [ 6674.913281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6674.915862] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6674.925203] LustreError: Skipped 1 previous similar message [ 6674.935782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6674.946221] Lustre: Skipped 2 previous similar messages [ 6674.970367] Lustre: Skipped 3 previous similar messages [ 6677.989151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6677.998799] Lustre: Skipped 1 previous similar message [ 6679.749917] Lustre: server umount lustre-OST0000 complete [ 6684.641048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6684.647483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6684.658298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6690.014778] Lustre: server umount lustre-MDT0000 complete [ 6693.351644] LustreError: 136145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6693.386990] LustreError: 136145:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6693.860747] LustreError: 136125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819187 with bad export cookie 9457286688626133939 [ 6693.862517] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6693.867656] LustreError: 136125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6694.207539] Lustre: server umount lustre-MDT0001 complete [ 6705.776066] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 6708.519716] Key type lgssc unregistered [ 6708.844384] LNet: 147216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6708.857501] LNetError: 147216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6709.930785] LNet: Removed LNI 192.168.202.151@tcp [ 6710.720172] Key type .llcrypt unregistered [ 6710.722514] Key type ._llcrypt unregistered [ 6730.416364] Key type ._llcrypt registered [ 6730.418466] Key type .llcrypt registered [ 6730.522441] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 6746.537527] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 6747.762543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6747.875450] alg: No test for adler32 (adler32-zlib) [ 6748.938711] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 6749.317228] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 6750.999412] Key type lgssc registered [ 6752.478827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6788.952310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6788.990051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6790.346437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6790.388881] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6790.496796] Lustre: lustre-MDT0000: new disk, initializing [ 6790.568068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6790.583471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6794.937149] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6807.404506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6807.521917] Lustre: 150415:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6807.573303] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6807.577444] Lustre: Skipped 1 previous similar message [ 6807.652895] Lustre: lustre-MDT0001: new disk, initializing [ 6807.808746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6807.881964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6807.896810] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6812.382057] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6821.129904] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6826.714932] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6834.705091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6835.087980] Lustre: lustre-OST0000: new disk, initializing [ 6835.091839] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6835.104432] Lustre: 152385:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6835.260201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6842.497392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6842.509814] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6842.511799] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6842.720807] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6852.274426] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6858.348934] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_module obdecho/obdecho [ 6860.322150] LustreError: 153388:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6860.328199] LustreError: 153388:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6862.077637] LustreError: 153485:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6878.707076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6878.711184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6878.731878] Lustre: Skipped 2 previous similar messages [ 6878.755653] Lustre: Skipped 2 previous similar messages [ 6881.508369] Lustre: server umount lustre-MDT0000 complete [ 6885.792817] LustreError: 152384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819379 with bad export cookie 13486868266385454744 [ 6885.794961] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6885.802530] LustreError: 152384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6886.329108] Lustre: server umount lustre-MDT0001 complete [ 6901.297277] Lustre: server umount lustre-OST0000 complete [ 6923.753227] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 6933.964467] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 6971.000637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6971.333973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6971.437922] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6971.566883] Lustre: lustre-MDT0000: new disk, initializing [ 6971.646074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6971.668568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6976.248319] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6986.721462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6986.854837] Lustre: 157200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6986.922045] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6986.925566] Lustre: Skipped 1 previous similar message [ 6986.988713] Lustre: lustre-MDT0001: new disk, initializing [ 6987.069731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6987.098463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6987.104469] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6991.120182] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6999.418325] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7005.557618] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7013.118430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7013.462620] Lustre: lustre-OST0000: new disk, initializing [ 7013.465779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7013.473359] Lustre: 159171:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7013.578830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7020.431607] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7023.229059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7023.255195] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7023.371695] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7031.156855] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7035.384502] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7035.648853] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7040.431447] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7040.648402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7044.075369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7044.075748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7044.078914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7044.086699] Lustre: Skipped 1 previous similar message [ 7044.111678] LustreError: Skipped 1 previous similar message [ 7048.825587] Lustre: server umount lustre-OST0000 complete [ 7053.805616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7053.811968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7053.825402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7053.827528] Lustre: Skipped 1 previous similar message [ 7059.435869] LustreError: 157212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7059.449710] LustreError: 157212:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7059.778971] Lustre: server umount lustre-MDT0000 complete [ 7064.551802] LustreError: 157206:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7064.587884] LustreError: 157206:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7064.612672] LustreError: 157997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819558 with bad export cookie 13486868266385455703 [ 7064.617332] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7064.627053] LustreError: 157997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7065.056670] Lustre: server umount lustre-MDT0001 complete [ 7069.780688] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 14:46:00 (1786819560) [ 7072.132292] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7074.267622] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 14:46:05 (1786819565) [ 7083.417342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7084.519094] LustreError: 161584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7084.672521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7089.874544] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7090.162250] LustreError: 161585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7094.244346] LustreError: 161584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7100.163615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7105.268186] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7112.778961] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7118.306945] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7124.625761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7125.138775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7125.146668] Lustre: Skipped 1 previous similar message [ 7131.757609] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7139.181308] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7145.577460] Lustre: server umount lustre-MDT0000 complete [ 7146.984380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7147.004500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7147.023630] Lustre: Skipped 1 previous similar message [ 7147.035346] LustreError: 161585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7147.065994] LustreError: 161585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7150.653028] LustreError: 162320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819644 with bad export cookie 13486868266385456585 [ 7150.659360] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7150.679046] LustreError: 162320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7155.192694] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7155.215687] Lustre: Skipped 2 previous similar messages [ 7155.222108] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7155.224866] Lustre: Skipped 2 previous similar messages [ 7157.137983] Lustre: server umount lustre-MDT0001 complete [ 7164.810767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7177.317555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7177.560743] LustreError: 165425:0:(llog.c:1655:llog_backup()) MGC192.168.202.151@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7177.567367] Lustre: 165425:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.151@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7180.704728] LustreError: 165430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7180.735950] LustreError: 165430:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 7215.721236] LustreError: 165431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7215.753396] LustreError: 165431:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [ 7238.111358] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7238.122241] Lustre: 165425:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.151@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7279.102308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7281.259282] LustreError: 165461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7281.276672] LustreError: 165461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 43 previous similar messages [ 7283.975594] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7284.216813] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7292.927565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7296.485748] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbb2afb3b5664d6ff [ 7296.514502] Lustre: MGC192.168.202.151@tcp: Connection restored to 0@lo (at 0@lo) [ 7301.881315] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7302.138839] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7309.419427] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7311.134467] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7316.453866] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7318.081711] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7332.837831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7332.847917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7332.866930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7332.896275] Lustre: Skipped 3 previous similar messages [ 7335.174546] Lustre: server umount lustre-MDT0000 complete [ 7338.598647] LustreError: 161564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819832 with bad export cookie 13486868266385463039 [ 7338.600130] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7338.607209] LustreError: 161564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7338.955693] Lustre: server umount lustre-MDT0001 complete [ 7352.908551] Lustre: server umount lustre-OST0000 complete [ 7374.515584] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 7385.865516] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 7426.367668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7426.811640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7426.840814] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7426.918635] Lustre: lustre-MDT0000: new disk, initializing [ 7427.012196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7427.014946] Lustre: Skipped 1 previous similar message [ 7427.058428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7432.821989] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7445.115940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7445.202455] Lustre: 170532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7445.398059] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7451.441578] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7461.359896] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7466.611758] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7473.320342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7473.666101] Lustre: lustre-OST0000: new disk, initializing [ 7473.678885] Lustre: Skipped 1 previous similar message [ 7473.686659] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7473.698575] Lustre: Skipped 2 previous similar messages [ 7473.710796] Lustre: 172501:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7480.452123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7480.457490] Lustre: Skipped 1 previous similar message [ 7480.460680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7480.549070] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7482.594148] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7495.107694] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7499.167696] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7499.392462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7504.224749] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7504.427552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7506.402212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7506.405862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7506.413246] LustreError: Skipped 1 previous similar message [ 7506.420483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7506.443392] Lustre: Skipped 3 previous similar messages [ 7512.624626] Lustre: server umount lustre-OST0000 complete [ 7522.748973] Lustre: server umount lustre-MDT0000 complete [ 7526.073079] LustreError: 170524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820019 with bad export cookie 13486868266385479881 [ 7526.087804] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7526.301662] Lustre: server umount lustre-MDT0001 complete [ 7534.873340] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 14:53:46 (1786820026) [ 7540.675977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7541.163802] LustreError: 174913:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7541.184969] LustreError: 174913:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 7545.223665] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7553.705871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7559.291581] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7568.996346] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7576.279659] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7583.612797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7593.253956] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7604.321712] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7617.519247] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 14:55:08 (1786820108) [ 7625.698137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7625.700534] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7625.723677] Lustre: Skipped 2 previous similar messages [ 7625.724945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7625.760631] Lustre: Skipped 8 previous similar messages [ 7628.505175] Lustre: server umount lustre-OST0000 complete [ 7639.012733] Lustre: server umount lustre-MDT0000 complete [ 7642.682250] LustreError: 177894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820136 with bad export cookie 13486868266385480756 [ 7642.688218] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7642.691647] LustreError: 177894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7643.010724] Lustre: server umount lustre-MDT0001 complete [ 7654.233477] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 7656.857484] Key type lgssc unregistered [ 7657.261799] LNet: 178649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7657.285193] LNetError: 178649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7657.315718] LNet: Removed LNI 192.168.202.151@tcp [ 7658.461499] Key type .llcrypt unregistered [ 7658.469965] Key type ._llcrypt unregistered [ 7684.003558] Key type ._llcrypt registered [ 7684.006529] Key type .llcrypt registered [ 7684.132873] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 7686.298123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7686.354594] alg: No test for adler32 (adler32-zlib) [ 7687.576946] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 7688.027295] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 7689.815342] Key type lgssc registered [ 7691.411769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7699.822763] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7699.847725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7715.427224] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7720.543866] LustreError: 179892:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.151@tcp: failed processing log, type 4: rc = -110 [ 7725.663733] LustreError: 179892:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7725.696119] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7725.706139] LustreError: MGC192.168.202.151@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7725.718317] LustreError: 179892:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7725.725644] LustreError: 179892:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7725.736306] LustreError: 179892:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7725.843460] Lustre: server umount lustre-OST0000 complete [ 7725.848712] LustreError: 179892:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7734.588081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7736.146861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7740.959668] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7745.505256] LustreError: 180391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7745.517547] LustreError: 180391:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7749.589939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7749.979536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7754.379410] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7761.254808] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7766.448304] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7772.856106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7773.044790] LustreError: 182207:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7773.050865] Lustre: 182207:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.151@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7773.217447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7779.951698] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7787.798307] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7790.562223] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7790.569640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7790.586738] Lustre: Skipped 2 previous similar messages [ 7790.591658] Lustre: Skipped 2 previous similar messages [ 7795.681449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7795.691135] Lustre: Skipped 2 previous similar messages [ 7800.804926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7800.812642] Lustre: Skipped 2 previous similar messages [ 7804.383425] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7804.649596] Lustre: server umount lustre-MDT0000 complete [ 7805.921520] LustreError: 181120:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7805.941447] LustreError: 181120:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7809.357616] LustreError: 180373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820303 with bad export cookie 15209578602731799505 [ 7809.359468] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7809.371487] LustreError: 180373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7809.805387] Lustre: server umount lustre-MDT0001 complete [ 7823.801759] Lustre: server umount lustre-OST0000 complete [ 7833.987221] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 7843.818494] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 7877.675963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7878.013839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7878.056435] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7878.128563] Lustre: lustre-MDT0000: new disk, initializing [ 7878.196181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7878.206530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7882.667464] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7895.478335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7895.776074] Lustre: 185732:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7895.880761] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7895.885065] Lustre: Skipped 1 previous similar message [ 7895.967632] Lustre: lustre-MDT0001: new disk, initializing [ 7896.132805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7896.180436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7896.207495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7901.836856] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7912.772968] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7919.953966] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7927.806584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7928.220934] Lustre: lustre-OST0000: new disk, initializing [ 7928.228091] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7928.244554] Lustre: 187700:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7928.371713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7936.156305] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7937.616615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7937.631974] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7937.721572] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7945.504855] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7950.426458] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7950.598478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7955.242635] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7955.573562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7958.497971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7958.507038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7958.508845] LustreError: Skipped 1 previous similar message [ 7958.514819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7958.531021] Lustre: Skipped 1 previous similar message [ 7963.630493] LustreError: 188144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7963.860152] Lustre: server umount lustre-OST0000 complete [ 7967.712045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7967.737622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7967.767306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7967.782048] Lustre: Skipped 1 previous similar message [ 7974.100107] Lustre: server umount lustre-MDT0000 complete [ 7977.918423] LustreError: 185725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820471 with bad export cookie 15209578602731800226 [ 7977.930526] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7978.321465] Lustre: server umount lustre-MDT0001 complete [ 7986.843740] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 15:01:18 (1786820478) [ 8003.994186] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 8006.813209] Key type lgssc unregistered [ 8007.073780] LNet: 190662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8007.078184] LNetError: 190662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8007.086841] LNet: Removed LNI 192.168.202.151@tcp [ 8007.839329] Key type .llcrypt unregistered [ 8007.842492] Key type ._llcrypt unregistered [ 8025.864961] Key type ._llcrypt registered [ 8025.867466] Key type .llcrypt registered [ 8025.990648] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 8027.328337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8027.358460] alg: No test for adler32 (adler32-zlib) [ 8028.633831] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 8028.980530] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 8030.631205] Key type lgssc registered [ 8031.834421] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8039.129851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8039.160850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8040.489028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8044.975348] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8050.657769] LustreError: 191879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8050.678128] LustreError: 191879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8053.384589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8053.842079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8058.831370] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8066.394880] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8071.869651] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8079.605171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8079.988578] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8079.995556] LustreError: 193743:0:(llog.c:1696:llog_backup()) MGC192.168.202.151@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8080.008060] Lustre: 193743:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.151@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8080.201442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8086.545873] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8093.625749] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8098.936230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8098.948214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8098.958679] Lustre: Skipped 2 previous similar messages [ 8098.970562] Lustre: Skipped 1 previous similar message [ 8102.233741] Lustre: server umount lustre-MDT0000 complete [ 8104.420496] LustreError: 191875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8104.445532] LustreError: 191875:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8105.718175] LustreError: 191859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820599 with bad export cookie 8724899561414532887 [ 8105.730385] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8107.167519] LustreError: 192608:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000089282673 x1873617115704960/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8107.352527] Lustre: server umount lustre-MDT0001 complete [ 8121.719184] Lustre: server umount lustre-OST0000 complete [ 8131.582424] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 8140.809136] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 8176.749389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8177.054915] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8177.133616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8177.230586] Lustre: lustre-MDT0000: new disk, initializing [ 8177.402986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8177.430348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8182.246623] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8192.573795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8192.662120] Lustre: 197269:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8192.697952] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8192.699878] Lustre: Skipped 1 previous similar message [ 8192.703463] LustreError: 198052:0:(llog.c:1696:llog_backup()) MGC192.168.202.151@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8192.709191] LustreError: 198052:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8192.713304] Lustre: 198052:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.151@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8192.727455] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8192.731128] Lustre: Skipped 1 previous similar message [ 8192.793534] Lustre: lustre-MDT0001: new disk, initializing [ 8192.872920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8192.908045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8192.920937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8196.918812] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8205.247920] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8210.531338] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8216.814116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8217.081828] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8217.087328] Lustre: Skipped 1 previous similar message [ 8217.092626] LustreError: 199214:0:(llog.c:1696:llog_backup()) MGC192.168.202.151@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8217.104156] LustreError: 199214:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8217.113381] Lustre: 199214:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.151@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8217.124706] Lustre: 199214:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8217.145952] Lustre: lustre-OST0000: new disk, initializing [ 8217.152644] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8217.160471] Lustre: 199237:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8217.243882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8222.628803] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8227.396930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8227.408612] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8227.546450] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8232.012327] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8236.252649] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8236.507977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8241.314677] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8241.494604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8243.182262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8243.188844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8243.205692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8243.213659] Lustre: Skipped 1 previous similar message [ 8243.226597] Lustre: Skipped 2 previous similar messages [ 8248.293326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8248.302295] Lustre: Skipped 1 previous similar message [ 8249.282797] Lustre: server umount lustre-OST0000 complete [ 8253.415039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8253.427849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8253.430937] Lustre: Skipped 1 previous similar message [ 8253.448343] Lustre: Skipped 1 previous similar message [ 8258.529926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8258.537487] Lustre: Skipped 1 previous similar message [ 8258.902908] Lustre: server umount lustre-MDT0000 complete [ 8262.606288] LustreError: 197259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820756 with bad export cookie 8724899561414533608 [ 8262.613154] LustreError: 197259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8262.617045] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8263.006179] Lustre: server umount lustre-MDT0001 complete [ 8272.816179] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 15:06:04 (1786820764) [ 8288.014218] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 8298.022737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8298.414540] LustreError: 202363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8298.484571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8302.369552] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8303.588235] LustreError: 202364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8308.711772] LustreError: 202363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8311.834820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8316.558311] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8319.137788] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8325.858577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8333.565449] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8342.078653] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8342.241730] Lustre: lustre-OST0001: new disk, initializing [ 8342.245508] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8342.250310] Lustre: 204676:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8342.334377] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8342.337488] Lustre: Skipped 2 previous similar messages [ 8347.749239] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8350.872214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8350.886174] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8351.002426] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8358.143505] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8368.525510] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8370.537591] Lustre: DEBUG MARKER: ADD OST3 [ 8375.874508] loop: module loaded [ 8375.929184] loop0: detected capacity change from 0 to 204800000 [ 8376.159791] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8376.186738] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8376.212135] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8382.201331] loop0: detected capacity change from 0 to 204800000 [ 8383.381654] Lustre: DEBUG MARKER: STOP OST3 [ 8388.427843] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8388.663973] Lustre: lustre-OST0003: new disk, initializing [ 8388.675936] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8388.690587] Lustre: 206663:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8390.839051] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8391.022502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8391.026947] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8391.070609] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8392.658639] Lustre: Failing over lustre-OST0003 [ 8392.863308] LustreError: 206875:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8392.891313] Lustre: server umount lustre-OST0003 complete [ 8395.415232] LustreError: 205418:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8395.457223] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8396.283417] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8396.296899] LustreError: Skipped 1 previous similar message [ 8396.299118] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8396.322631] Lustre: Skipped 1 previous similar message [ 8403.720071] Lustre: DEBUG MARKER: START OST3 again [ 8409.804184] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8410.184636] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8410.193643] Lustre: Skipped 1 previous similar message [ 8410.215884] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8410.731270] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8411.982699] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8411.982768] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8416.152407] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8435.399912] Lustre: Failing over lustre-OST0003 [ 8435.494600] Lustre: server umount lustre-OST0003 complete [ 8435.687054] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8435.702045] LustreError: 208792:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8435.708541] Lustre: Skipped 1 previous similar message [ 8435.732624] LustreError: 208792:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 8455.649104] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8455.654427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8455.670928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8458.914449] Lustre: server umount lustre-MDT0000 complete [ 8462.627571] LustreError: 204675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820956 with bad export cookie 8724899561414534490 [ 8462.631908] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8462.637821] LustreError: 204675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8469.142936] Lustre: server umount lustre-MDT0001 complete [ 8473.382868] Lustre: server umount lustre-OST0000 complete [ 8477.298716] Lustre: server umount lustre-OST0001 complete [ 8484.482386] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 8493.042199] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 8536.569150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8537.051522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8537.092025] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8537.239700] Lustre: lustre-MDT0000: new disk, initializing [ 8537.426846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8541.900768] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8551.795861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8551.940721] Lustre: 212133:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8552.162491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8552.169261] Lustre: Skipped 1 previous similar message [ 8552.201864] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8556.469915] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8566.110897] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8572.345929] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8579.193149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8579.492780] Lustre: lustre-OST0000: new disk, initializing [ 8579.499824] Lustre: Skipped 1 previous similar message [ 8579.508444] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8579.515214] Lustre: Skipped 2 previous similar messages [ 8579.524497] Lustre: 214101:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8581.244024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8581.253054] Lustre: Skipped 1 previous similar message [ 8581.261231] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8581.363162] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8586.116314] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8595.994598] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8600.846806] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8601.081223] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8606.055772] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8606.271207] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8612.323534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8612.326107] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8612.338839] Lustre: Skipped 5 previous similar messages [ 8612.342527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8612.348797] LustreError: Skipped 1 previous similar message [ 8612.372701] Lustre: Skipped 9 previous similar messages [ 8614.267811] Lustre: server umount lustre-OST0000 complete [ 8618.980812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8624.591811] Lustre: server umount lustre-MDT0000 complete [ 8627.684595] LustreError: 212145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8627.701757] LustreError: 212145:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 8627.976884] LustreError: 215642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821121 with bad export cookie 8724899561414536786 [ 8627.980364] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8627.997579] LustreError: 215642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8628.389725] Lustre: server umount lustre-MDT0001 complete [ 8631.670220] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 15:12:03 (1786821123) [ 8647.875377] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 8658.539311] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 8694.975081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8695.262921] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8695.295781] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8695.401908] Lustre: lustre-MDT0000: new disk, initializing [ 8695.501936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8700.174390] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8710.230229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8710.324804] Lustre: 218711:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8710.592336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8715.134295] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8724.315490] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8730.874660] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8737.607751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8737.928783] Lustre: 220682:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8739.831674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8743.248883] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8751.999419] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8756.102585] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8756.255607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8760.619708] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8760.785720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8765.412975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8765.416228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8765.423443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8765.423451] Lustre: Skipped 3 previous similar messages [ 8765.455730] Lustre: Skipped 4 previous similar messages [ 8768.618620] Lustre: server umount lustre-OST0000 complete [ 8778.748680] Lustre: server umount lustre-MDT0000 complete [ 8780.772250] LustreError: 218719:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8780.785717] LustreError: 218719:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8782.689291] LustreError: 222222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821276 with bad export cookie 8724899561414537668 [ 8782.697495] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8782.708855] LustreError: 222222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8783.134753] Lustre: server umount lustre-MDT0001 complete [ 8791.125946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8795.710571] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8804.918323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8809.591445] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8817.878209] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8822.922661] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8829.483554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8829.756380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8829.762757] Lustre: Skipped 6 previous similar messages [ 8835.435694] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8841.617856] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8851.426386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8851.435365] LustreError: Skipped 1 previous similar message [ 8851.440821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8851.465601] Lustre: Skipped 2 previous similar messages [ 8851.479805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8851.487036] Lustre: Skipped 3 previous similar messages [ 8857.446878] Lustre: server umount lustre-OST0000 complete [ 8867.838520] Lustre: server umount lustre-MDT0000 complete [ 8871.644296] LustreError: 222878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821365 with bad export cookie 8724899561414538543 [ 8871.658934] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8871.660184] LustreError: 222878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8872.155712] Lustre: server umount lustre-MDT0001 complete [ 8983.406928] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 8986.298974] Key type lgssc unregistered [ 8986.683794] LNet: 226570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8986.699625] LNetError: 226570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8986.718065] LNet: Removed LNI 192.168.202.151@tcp [ 8987.568211] Key type .llcrypt unregistered [ 8987.573079] Key type ._llcrypt unregistered [ 9012.587106] Key type ._llcrypt registered [ 9012.592111] Key type .llcrypt registered [ 9012.836055] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 9028.360609] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 9029.626953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9029.690072] alg: No test for adler32 (adler32-zlib) [ 9030.944737] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 9031.230432] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 9032.960899] Key type lgssc registered [ 9034.418575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9072.902407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9072.939669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9074.287374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9074.336868] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9074.421496] Lustre: lustre-MDT0000: new disk, initializing [ 9074.516784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9074.553119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9079.472792] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9089.528550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9089.651447] Lustre: 229924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9089.690745] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9089.693941] Lustre: Skipped 1 previous similar message [ 9089.744775] Lustre: lustre-MDT0001: new disk, initializing [ 9089.819254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9089.860725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9089.882173] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9094.761647] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9104.495906] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9110.521768] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9118.342704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9118.698580] Lustre: lustre-OST0000: new disk, initializing [ 9118.702852] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9118.708474] Lustre: 231890:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9118.805063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9126.425848] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9129.086606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9129.110373] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9129.213907] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9136.642579] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9142.701550] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9142.988693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9147.900369] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9148.179540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9155.041878] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9155.050369] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9155.071643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9155.077060] Lustre: Skipped 1 previous similar message [ 9156.727517] Lustre: server umount lustre-OST0000 complete [ 9161.697280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9161.713851] LustreError: Skipped 1 previous similar message [ 9161.716910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9161.751205] Lustre: Skipped 1 previous similar message [ 9161.769160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9165.315712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9165.324814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9167.482942] Lustre: server umount lustre-MDT0000 complete [ 9171.269655] LustreError: 229915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821665 with bad export cookie 6490918834984308004 [ 9171.270919] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9171.290227] LustreError: 229915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9171.723657] Lustre: server umount lustre-MDT0001 complete [ 9190.508441] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 9202.389468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9202.691885] LustreError: 234825:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9202.708044] LustreError: 234825:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9202.741239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9207.777827] LustreError: 234826:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9207.847591] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9212.904357] LustreError: 234825:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9217.390160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9217.819675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9222.914946] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9226.031181] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9232.629646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9232.962895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9239.289994] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9247.494073] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9247.670738] Lustre: lustre-OST0001: new disk, initializing [ 9247.673971] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9247.679580] Lustre: 237131:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9253.385568] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9255.574283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9255.583301] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9255.649665] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9265.325580] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9274.446354] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9289.965796] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 15:23:00 (1786821780) [ 9294.818198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9294.831947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9294.877317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9294.892560] Lustre: Skipped 1 previous similar message [ 9298.850647] Lustre: server umount lustre-MDT0000 complete [ 9301.993448] LustreError: 234822:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9302.008145] LustreError: 234822:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9302.913164] LustreError: 234808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821796 with bad export cookie 6490918834984308886 [ 9302.922532] LustreError: 234808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9302.924869] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9303.547108] Lustre: server umount lustre-MDT0001 complete [ 9317.423112] Lustre: server umount lustre-OST0000 complete [ 9332.378953] Lustre: server umount lustre-OST0001 complete [ 9339.603362] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 9349.158536] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 9384.531745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9384.790548] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9384.840201] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9384.950426] Lustre: lustre-MDT0000: new disk, initializing [ 9385.062059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9385.067158] Lustre: Skipped 1 previous similar message [ 9385.084359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9389.921770] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9401.892371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9402.022051] Lustre: 241078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9402.066756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9402.069252] Lustre: Skipped 1 previous similar message [ 9402.119433] Lustre: lustre-MDT0001: new disk, initializing [ 9402.233392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9402.245147] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9406.492647] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9416.137838] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9422.193540] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9429.878375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9430.208259] Lustre: lustre-OST0000: new disk, initializing [ 9430.212351] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9430.228158] Lustre: 243049:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9430.342534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9430.345310] Lustre: Skipped 1 previous similar message [ 9431.393056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9431.426961] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9431.606881] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9437.761744] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9449.108855] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9454.227281] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9454.477131] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9459.242221] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9459.532230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9462.240824] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9462.245317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9462.254115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9462.254123] Lustre: Skipped 4 previous similar messages [ 9462.283977] Lustre: Skipped 4 previous similar messages [ 9467.661426] Lustre: server umount lustre-OST0000 complete [ 9472.481936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9472.490048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9472.490498] Lustre: Skipped 1 previous similar message [ 9472.501928] Lustre: Skipped 4 previous similar messages [ 9477.487454] Lustre: server umount lustre-MDT0000 complete [ 9477.599889] LustreError: 241091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9477.618356] LustreError: 241091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9480.790945] LustreError: 241073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821974 with bad export cookie 6490918834984311021 [ 9480.795492] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9480.800762] LustreError: 241073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9481.122487] Lustre: server umount lustre-MDT0001 complete [ 9488.835674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9489.350230] LustreError: 245264:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9494.068611] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9502.937627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9503.294660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9503.302881] Lustre: Skipped 1 previous similar message [ 9507.813246] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9516.330685] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9523.134894] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9532.127194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9538.465497] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9547.062768] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9697.249790] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9697.250842] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9697.295214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9697.304727] Lustre: Skipped 1 previous similar message [ 9700.962320] Lustre: server umount lustre-OST0000 complete [ 9711.566662] Lustre: server umount lustre-MDT0000 complete [ 9712.615138] LustreError: 245837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9712.644634] LustreError: 245837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9715.312161] LustreError: 245246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822209 with bad export cookie 6490918834984311903 [ 9715.313908] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9715.324905] LustreError: 245246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9715.875273] Lustre: server umount lustre-MDT0001 complete [ 9815.297538] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 9818.470924] Key type lgssc unregistered [ 9818.891091] LNet: 248877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9818.903886] LNetError: 248877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9818.917173] LNet: Removed LNI 192.168.202.151@tcp [ 9820.073194] Key type .llcrypt unregistered [ 9820.079379] Key type ._llcrypt unregistered [ 9848.128797] Key type ._llcrypt registered [ 9848.130414] Key type .llcrypt registered [ 9848.238166] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 9864.324621] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 9865.154889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9865.245651] alg: No test for adler32 (adler32-zlib) [ 9866.350621] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 9866.616867] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 9868.327169] Key type lgssc registered [ 9869.658396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9916.320622] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 9926.030499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9926.056364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9927.307051] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9927.335927] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9927.422663] Lustre: lustre-MDT0000: new disk, initializing [ 9927.517373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9927.539719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9931.642202] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9942.625502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9942.765393] Lustre: 252950:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9942.813985] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9942.816844] Lustre: Skipped 1 previous similar message [ 9942.895851] Lustre: lustre-MDT0001: new disk, initializing [ 9942.986703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9943.044120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9943.056322] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9947.139126] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9951.972625] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9959.256572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9959.592180] Lustre: lustre-OST0000: new disk, initializing [ 9959.595810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9959.607793] Lustre: 254584:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9959.752175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9963.478491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9963.492798] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9963.620772] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9966.167291] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9977.410903] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9977.732592] Lustre: lustre-OST0001: new disk, initializing [ 9977.736558] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9977.742976] Lustre: 255455:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9977.825890] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9983.984236] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9986.645755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9986.660520] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9986.765799] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9994.001111] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10005.424805] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10017.767378] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10017.770796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10017.782800] Lustre: Skipped 1 previous similar message [10017.791221] Lustre: Skipped 3 previous similar messages [10022.563646] Lustre: server umount lustre-MDT0000 complete [10026.259156] LustreError: 253746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822520 with bad export cookie 3129161485560880652 [10026.261746] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10026.288692] LustreError: 253746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10026.658324] Lustre: server umount lustre-MDT0001 complete [10039.365709] Lustre: server umount lustre-OST0000 complete [10053.091751] Lustre: server umount lustre-OST0001 complete [10066.981050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10078.128215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10088.551073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10099.362160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10119.173421] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [10130.800550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10130.836948] Lustre: lustre-MDT0000: reset Object Index mappings [10131.334106] LustreError: 260465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10131.351425] LustreError: 260465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10131.439386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10136.555272] LustreError: 260466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10137.682635] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10141.685304] LustreError: 260465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10146.791387] LustreError: 260466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10148.280352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10148.293290] Lustre: lustre-MDT0001: reset Object Index mappings [10148.897960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10154.302548] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10157.705809] Lustre: 261611:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10165.033355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10165.398623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10173.622273] LustreError: 261965:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10173.794098] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10176.718754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10182.118262] LustreError: 261965:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10182.155826] LustreError: 261965:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10182.743161] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10188.289724] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10189.449861] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10197.177535] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10200.628970] Lustre: 263481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10206.965188] Lustre: server umount lustre-MDT0000 complete [10208.740123] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10208.746294] LustreError: 260460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10208.759505] Lustre: Skipped 4 previous similar messages [10208.792794] LustreError: 260460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10211.868304] LustreError: 261213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822705 with bad export cookie 3129161485560887442 [10211.878433] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10211.883435] LustreError: 261213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10212.351930] Lustre: server umount lustre-MDT0001 complete [10227.262833] Lustre: server umount lustre-OST0000 complete [10230.111107] Lustre: 250456:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786822707/real 1786822707] req@ffffa0dbbeb1b100 x1873619043087488/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786822723 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10230.129810] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10230.139792] Lustre: Skipped 1 previous similar message [10231.414259] Lustre: server umount lustre-OST0001 complete [10253.420384] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 15:39:04 (1786822744) [10272.151461] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [10282.066698] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [10319.249808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10319.525317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10319.555404] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10319.642460] Lustre: lustre-MDT0000: new disk, initializing [10319.735448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10319.742739] Lustre: Skipped 1 previous similar message [10319.763833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10324.277364] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10334.311567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10334.438750] Lustre: 267734:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10334.483979] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10334.487594] Lustre: Skipped 1 previous similar message [10334.550609] Lustre: lustre-MDT0001: new disk, initializing [10334.673412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10334.682715] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10339.438629] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10351.810846] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10360.020686] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10366.811181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10367.084445] Lustre: lustre-OST0000: new disk, initializing [10367.087957] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10367.093406] Lustre: 269705:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10368.611723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10368.619862] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10368.687235] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10373.804909] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10383.337590] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10388.881609] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10389.101720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10393.953636] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10394.254103] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10399.722518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10399.730352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10399.745335] LustreError: Skipped 1 previous similar message [10399.747441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10399.768325] Lustre: Skipped 1 previous similar message [10402.792569] Lustre: server umount lustre-OST0000 complete [10409.958259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10409.966542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10409.978753] Lustre: Skipped 1 previous similar message [10409.993682] Lustre: Skipped 2 previous similar messages [10413.767684] Lustre: server umount lustre-MDT0000 complete [10415.077531] LustreError: 267745:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10415.093202] LustreError: 267745:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10417.160069] LustreError: 268530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822910 with bad export cookie 3129161485560889535 [10417.179237] LustreError: 268530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10417.181945] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10417.627357] Lustre: server umount lustre-MDT0001 complete [10438.097948] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [10447.803995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10448.134174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10448.142434] Lustre: Skipped 2 previous similar messages [10452.936300] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10462.177543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10468.635453] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10472.516571] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10480.972867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10488.359229] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10496.895906] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10497.078727] Lustre: lustre-OST0001: new disk, initializing [10497.082905] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10497.088425] Lustre: 274951:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10498.541662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10498.554028] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10498.698741] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10504.425169] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10516.237603] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10520.756660] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10526.376504] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10526.452282] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10534.380380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10534.398391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10534.398712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10534.403831] Lustre: Skipped 3 previous similar messages [10534.418000] Lustre: Skipped 3 previous similar messages [10539.492256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10539.999506] Lustre: server umount lustre-MDT0000 complete [10543.874351] LustreError: 272622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823037 with bad export cookie 3129161485560890417 [10543.877324] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10543.883426] LustreError: 272622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10544.210900] Lustre: server umount lustre-MDT0001 complete [10558.612678] Lustre: server umount lustre-OST0000 complete [10560.993654] Lustre: 250457:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786823038/real 1786823038] req@ffffa0dbb69d3800 x1873619043196544/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786823054 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10561.023745] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10564.008938] Lustre: server umount lustre-OST0001 complete [10579.572405] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [10589.214693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10589.519962] LustreError: 278009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10589.537912] LustreError: 278009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10589.605547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10589.612069] Lustre: Skipped 3 previous similar messages [10593.973393] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10603.787500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10610.493797] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10614.375205] Lustre: 279149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10622.614839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10632.881177] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10634.815616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10645.018495] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10651.143542] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10651.863859] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10660.301128] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10664.617454] Lustre: 281026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10671.386225] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10671.710083] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10686.850870] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 15:46:17 (1786823177) [10706.411468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10706.440408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10706.462127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10706.478773] Lustre: Skipped 3 previous similar messages [10709.624382] Lustre: server umount lustre-MDT0000 complete [10714.149081] LustreError: 281736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823207 with bad export cookie 3129161485560896969 [10714.161986] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10714.162753] LustreError: 281736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10717.665153] LustreError: 278004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10717.688180] LustreError: 278004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [10720.647028] Lustre: server umount lustre-MDT0001 complete [10730.611256] Lustre: server umount lustre-OST0000 complete [10734.728953] Lustre: server umount lustre-OST0001 complete [10741.192465] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [10751.170706] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [10801.527330] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [10815.565747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10815.866547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10815.923955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10816.020227] Lustre: lustre-MDT0000: new disk, initializing [10816.145976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10823.297323] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10837.828689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10838.100041] Lustre: 285273:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10838.540921] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10844.205141] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10849.078220] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10856.997309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10857.341112] Lustre: 286903:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10857.459201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10857.465388] Lustre: Skipped 5 previous similar messages [10859.165712] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10864.135379] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10875.900337] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10876.164842] Lustre: 287775:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10878.083690] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10878.091665] Lustre: Skipped 1 previous similar message [10878.293045] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10882.613220] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10892.588644] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10912.920519] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 15:50:03 (1786823403) [10915.297788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10915.304827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10915.315262] Lustre: Skipped 5 previous similar messages [10915.327564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10915.333401] Lustre: Skipped 8 previous similar messages [10917.668054] Lustre: server umount lustre-MDT0000 complete [10921.183083] LustreError: 286077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823414 with bad export cookie 3129161485560904606 [10921.197724] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10921.207123] LustreError: 286077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10921.538770] Lustre: server umount lustre-MDT0001 complete [10936.602679] Lustre: server umount lustre-OST0000 complete [10940.907391] Lustre: 250456:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786823418/real 1786823418] req@ffffa0da8be2a680 x1873619043356928/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786823434 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10941.391440] Lustre: server umount lustre-OST0001 complete [10950.826820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10956.553990] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10965.610464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10970.106483] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10977.293901] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10983.363806] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10990.805677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10996.356557] LustreError: 291857:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10996.411275] LustreError: 291857:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [10999.530551] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11010.109044] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11030.474213] loop1: detected capacity change from 0 to 204800000 [11030.677805] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11030.684216] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11030.694330] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11035.795362] loop1: detected capacity change from 0 to 204800000 [11041.606043] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11041.697540] LustreError: 290015:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs30274: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11041.709756] LustreError: 290015:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs30274 [11041.714847] LustreError: 290015:0:(mgs_handler.c:517:mgs_target_reg()) fs30274-MDT0001: Error checking target index: rc = -13 [11041.723241] LustreError: fs30274-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11041.729504] LustreError: 293744:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11041.737274] LustreError: 293744:0:(tgt_mount.c:2129:server_put_super()) no obd fs30274-MDT0001 [11041.740359] LustreError: 293744:0:(tgt_mount.c:116:server_deregister_mount()) fs30274-MDT0001 not registered [11041.791356] Lustre: server umount fs30274-MDT0001 complete [11041.795261] LustreError: 293744:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11050.363963] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11050.447181] Lustre: Setting parameter fs30274-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs30274-MDT0001 [11050.458034] Lustre: Skipped 1 previous similar message [11050.497864] Lustre: srv-fs30274-MDT0001: No data found on store. Initialize space. [11050.500890] Lustre: Skipped 4 previous similar messages [11050.570669] Lustre: fs30274-MDT0001: new disk, initializing [11050.579373] Lustre: Skipped 3 previous similar messages [11050.671984] Lustre: srv-fs30274-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11050.679485] Lustre: 294211:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs30274-MDT0001-mdtlov: can't create new llog: rc = -115 [11050.690106] LustreError: 294211:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs30274-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11055.371922] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11063.423226] Lustre: server umount fs30274-MDT0001 complete [11077.646693] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11080.592660] loop1: detected capacity change from 0 to 204800000 [11086.645188] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11087.086264] Lustre: 295781:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs30274-MDT0001-mdtlov: can't create new llog: rc = -115 [11087.099537] LustreError: 295781:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs30274-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11091.958440] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11092.972656] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11097.051616] Lustre: server umount fs30274-MDT0001 complete [11108.009254] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.51@tcp (stopping) [11108.336612] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11108.356330] Lustre: Skipped 5 previous similar messages [11112.919195] Lustre: server umount lustre-MDT0000 complete [11121.255746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11121.384944] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11126.243841] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11126.771131] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11134.309439] loop1: detected capacity change from 0 to 204800000 [11140.910131] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11141.487574] Lustre: 298226:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs30274-MDT0001-mdtlov: can't create new llog: rc = -115 [11141.502397] LustreError: 298226:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs30274-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11146.687729] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11153.164764] Lustre: server umount fs30274-MDT0001 complete [11177.982756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11177.985042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11178.004769] Lustre: Skipped 4 previous similar messages [11192.287178] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11192.449348] Lustre: server umount lustre-OST0000 complete [11203.089836] Lustre: server umount lustre-MDT0000 complete [11207.188522] LustreError: 294755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823700 with bad export cookie 3129161485560907287 [11207.200306] LustreError: 294755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11207.203662] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11207.748398] Lustre: server umount lustre-MDT0001 complete [11220.244396] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [11223.540716] Key type lgssc unregistered [11223.824871] LNet: 300576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11223.833091] LNetError: 300576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11223.853914] LNet: Removed LNI 192.168.202.151@tcp [11224.986345] Key type .llcrypt unregistered [11224.991017] Key type ._llcrypt unregistered [11247.142681] Key type ._llcrypt registered [11247.144060] Key type .llcrypt registered [11247.230298] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [11248.457990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11248.478100] alg: No test for adler32 (adler32-zlib) [11249.486806] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [11249.778322] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [11251.489168] Key type lgssc registered [11252.848518] Lustre: Echo OBD driver; http://www.lustre.org/ [11262.023558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11262.064201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11263.668549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11268.920510] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11274.211393] LustreError: 301848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11274.224863] LustreError: 301848:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11278.729818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11279.305571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11285.242699] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11293.020580] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11298.973912] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11306.605234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11306.892476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11309.991875] LustreError: 303679:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11312.655470] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11315.178409] LustreError: 303681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11320.294526] LustreError: 303861:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11321.180911] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11328.127122] LustreError: 303861:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11328.149935] LustreError: 303861:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11338.359174] LustreError: 303861:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11338.368226] LustreError: 303861:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11342.577629] loop1: detected capacity change from 0 to 204800000 [11342.944948] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11342.966395] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11342.981675] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11349.847593] loop1: detected capacity change from 0 to 204800000 [11356.114557] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11356.425455] Lustre: fs15522-OST0000: new disk, initializing [11356.439560] Lustre: srv-fs15522-OST0000: No data found on store. Initialize space. [11356.450337] Lustre: 305582:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11356.585414] Lustre: fs15522-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11358.827798] LustreError: 303681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11358.869702] LustreError: 303681:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [11363.002462] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11373.878275] Lustre: server umount fs15522-OST0000 complete [11389.246183] loop1: detected capacity change from 0 to 204800000 [11389.937316] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11394.685159] LustreError: 303680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11394.706242] LustreError: 303680:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 20 previous similar messages [11395.768357] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11396.071865] Lustre: fs15522-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11402.726036] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11405.287793] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11411.327685] loop2: detected capacity change from 0 to 204800000 [11411.507527] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11411.518833] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11411.533689] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11416.195397] loop2: detected capacity change from 0 to 204800000 [11421.190323] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11421.305069] LustreError: fs15522-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11421.321978] LustreError: fs15522-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11421.341039] LustreError: 308623:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11421.347565] LustreError: 308623:0:(tgt_mount.c:2129:server_put_super()) no obd fs15522-OST0001 [11421.353941] LustreError: 308623:0:(tgt_mount.c:116:server_deregister_mount()) fs15522-OST0001 not registered [11421.410138] Lustre: server umount fs15522-OST0001 complete [11421.415682] LustreError: 308623:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11431.838636] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11431.959596] Lustre: fs15522-OST0001: new disk, initializing [11431.964320] Lustre: srv-fs15522-OST0001: No data found on store. Initialize space. [11431.969686] Lustre: 309176:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11432.062937] Lustre: fs15522-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11437.638631] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11446.257551] Lustre: server umount fs15522-OST0000 complete [11456.201643] Lustre: server umount fs15522-OST0001 complete [11459.555637] LustreError: 303680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11459.566698] LustreError: 303680:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 39 previous similar messages [11469.228279] loop1: detected capacity change from 0 to 204800000 [11475.395796] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11475.613257] Lustre: fs15522-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11482.019542] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11488.199552] loop2: detected capacity change from 0 to 204800000 [11494.868579] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11502.547907] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11512.324790] Lustre: server umount fs15522-OST0000 complete [11522.280841] Lustre: server umount fs15522-OST0001 complete [11538.404568] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11538.411245] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11538.419088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11540.456167] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11540.463640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11540.497249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11542.670229] Lustre: server umount lustre-OST0000 complete [11550.693740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11550.701277] LustreError: Skipped 1 previous similar message [11550.706826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11550.729553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11550.732608] Lustre: Skipped 2 previous similar messages [11553.625409] Lustre: server umount lustre-MDT0000 complete [11557.464911] LustreError: 301831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824051 with bad export cookie 14801181984982361714 [11557.481522] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11557.948737] Lustre: server umount lustre-MDT0001 complete [11570.652195] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [11573.354820] Key type lgssc unregistered [11573.587806] LNet: 315116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11573.593908] LNetError: 315116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11573.610689] LNet: Removed LNI 192.168.202.151@tcp [11574.690281] Key type .llcrypt unregistered [11574.691957] Key type ._llcrypt unregistered [11609.180520] Key type ._llcrypt registered [11609.182949] Key type .llcrypt registered [11609.272514] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [11624.601941] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [11625.858215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11626.068356] alg: No test for adler32 (adler32-zlib) [11627.156932] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [11627.407806] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [11629.063208] Key type lgssc registered [11630.228400] Lustre: Echo OBD driver; http://www.lustre.org/ [11674.327621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11674.403833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11675.729152] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11675.783459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11675.872939] Lustre: lustre-MDT0000: new disk, initializing [11675.965182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11675.984429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11681.507643] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11695.018872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11695.346209] Lustre: 318976:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11695.504449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11695.521774] Lustre: Skipped 1 previous similar message [11695.696971] Lustre: lustre-MDT0001: new disk, initializing [11695.900881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11695.984839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11695.999543] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11700.951868] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11711.811556] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11718.513759] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11725.475914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11725.942592] Lustre: lustre-OST0000: new disk, initializing [11725.958665] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11725.967822] Lustre: 320948:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11726.150650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11732.579753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11732.590495] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11732.720412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11732.887850] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11742.899471] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11745.192943] LustreError: 317148:0:(lmv_obd.c:211:lmv_notify()) activation of lustre-MDT0000_UUID failed: -22 [11745.249900] Lustre: Mounted lustre-client - version 2.17.57_2_gb7122c1 [11746.271982] Lustre: Unmounted lustre-client [11755.495018] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11755.503886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11755.507183] Lustre: Skipped 1 previous similar message [11755.521504] Lustre: Skipped 1 previous similar message [11760.628680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11760.640153] Lustre: Skipped 1 previous similar message [11761.838630] Lustre: server umount lustre-OST0000 complete [11767.777065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11767.796104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11767.817720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11770.850863] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11770.852380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11770.878217] Lustre: Skipped 1 previous similar message [11772.984056] Lustre: server umount lustre-MDT0000 complete [11776.936425] LustreError: 319771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824270 with bad export cookie 2264075984631404904 [11776.947838] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11776.964143] LustreError: 319771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11777.355312] Lustre: server umount lustre-MDT0001 complete [11789.803720] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [11794.280432] Key type lgssc unregistered [11796.774151] LNet: 323109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11796.783377] LNetError: 323109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11875.758713] LNet: Removed LNI 192.168.202.151@tcp [11876.584393] Key type .llcrypt unregistered [11876.586928] Key type ._llcrypt unregistered [11895.806884] Key type ._llcrypt registered [11895.809306] Key type .llcrypt registered [11895.977700] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [11915.886806] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [11917.132460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11917.173967] alg: No test for adler32 (adler32-zlib) [11918.500619] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [11918.912886] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [11920.711210] Key type lgssc registered [11922.573746] Lustre: Echo OBD driver; http://www.lustre.org/ [11955.631747] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 16:07:26 (1786824446) [11964.475498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11964.500604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11965.731692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11965.759731] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11965.825809] Lustre: lustre-MDT0000: new disk, initializing [11965.904729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11965.923734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11970.539557] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11981.424629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11981.564857] Lustre: 326333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11981.677622] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11981.685199] Lustre: Skipped 1 previous similar message [11981.818548] Lustre: lustre-MDT0001: new disk, initializing [11981.944888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11982.036741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11982.053597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11986.957310] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11997.358448] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12003.715771] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12010.360696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12010.668432] Lustre: lustre-OST0000: new disk, initializing [12010.671772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12010.677970] Lustre: 328301:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12010.794031] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12018.336563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12018.353683] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12018.636167] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12018.692128] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12030.564299] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12049.383662] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12049.387156] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12049.387883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12049.406599] Lustre: Skipped 1 previous similar message [12054.506646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12054.510757] Lustre: Skipped 2 previous similar messages [12055.393809] Lustre: server umount lustre-OST0000 complete [12059.617553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12059.627064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12059.634948] Lustre: Skipped 1 previous similar message [12059.646498] Lustre: Skipped 1 previous similar message [12064.747452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12064.753858] Lustre: Skipped 1 previous similar message [12065.383458] Lustre: server umount lustre-MDT0000 complete [12068.694438] LustreError: 326324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824562 with bad export cookie 1605134606073215573 [12068.702548] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12068.710503] LustreError: 326324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12069.096684] Lustre: server umount lustre-MDT0001 complete [12079.879117] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [12082.659538] Key type lgssc unregistered [12082.893675] LNet: 330214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12082.897922] LNetError: 330214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12083.944343] LNet: Removed LNI 192.168.202.151@tcp [12085.182799] Key type .llcrypt unregistered [12085.187895] Key type ._llcrypt unregistered [12105.633230] Key type ._llcrypt registered [12105.638361] Key type .llcrypt registered [12105.738269] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [12107.006617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12107.079343] alg: No test for adler32 (adler32-zlib) [12108.113345] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [12108.376742] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [12110.095541] Key type lgssc registered [12111.463674] Lustre: Echo OBD driver; http://www.lustre.org/ [12119.548031] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12119.591836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12121.295352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12126.588655] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12131.812483] LustreError: 331487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12131.832127] LustreError: 331487:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12136.349591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12136.789271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12142.254681] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12150.186648] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12155.812590] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12163.015702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12163.584908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12169.676263] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12177.762120] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12192.929732] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12195.391586] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12201.099993] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.51@tcp (stopping) [12202.465513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12202.477652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12202.486970] Lustre: Skipped 1 previous similar message [12202.500771] Lustre: Skipped 1 previous similar message [12206.184826] Lustre: server umount lustre-MDT0000 complete [12206.189397] LustreError: 333783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12207.597857] LustreError: 331487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12207.645718] LustreError: 331487:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12211.308166] LustreError: 331484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12212.176281] LustreError: 331467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824705 with bad export cookie 12049140353613503056 [12212.194721] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12212.202438] LustreError: 331467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12212.729794] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12212.771653] Lustre: Skipped 1 previous similar message [12212.795097] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12212.810791] Lustre: Skipped 1 previous similar message [12216.428740] LustreError: 331483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12216.486157] LustreError: 331483:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12216.488277] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.51@tcp (stopping) [12219.059981] Lustre: server umount lustre-MDT0001 complete [12225.281142] Lustre: server umount lustre-OST0000 complete [12235.019052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12240.539563] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12243.407556] Lustre: 335670:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.151@tcp [12245.559122] Lustre: server umount MGS complete [12254.115302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12254.538845] LustreError: 336242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12254.557994] LustreError: 336242:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12254.636251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12259.517269] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12268.403569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12268.775347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12273.617134] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12284.471306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12284.949833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12292.399798] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12301.051278] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12303.095589] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12308.999884] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12311.226785] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12317.159117] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12317.179296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12317.207134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12317.213320] Lustre: Skipped 1 previous similar message [12318.950900] Lustre: server umount lustre-OST0000 complete [12324.833662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12324.851034] Lustre: Skipped 1 previous similar message [12329.234302] Lustre: server umount lustre-MDT0000 complete [12329.956321] LustreError: 336242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12329.976954] LustreError: 336242:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [12333.479673] LustreError: 336224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824827 with bad export cookie 12049140353613503896 [12333.488966] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12333.492647] LustreError: 336224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12343.924615] Lustre: server umount lustre-MDT0001 complete [12356.027620] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [12359.358263] Key type lgssc unregistered [12359.762734] LNet: 339682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12359.769879] LNetError: 339682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12359.795938] LNet: Removed LNI 192.168.202.151@tcp [12360.856447] Key type .llcrypt unregistered [12360.858952] Key type ._llcrypt unregistered [12380.842201] Key type ._llcrypt registered [12380.844906] Key type .llcrypt registered [12380.994769] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [12397.080587] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [12398.569802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12398.586406] alg: No test for adler32 (adler32-zlib) [12399.597689] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [12399.758463] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [12401.399598] Key type lgssc registered [12402.810921] Lustre: Echo OBD driver; http://www.lustre.org/ [12440.263164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12440.301829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12441.658729] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12441.676452] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12441.787526] Lustre: lustre-MDT0000: new disk, initializing [12441.880964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12441.894401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12447.278499] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12459.318025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12459.475124] Lustre: 342848:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12459.585763] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12459.592097] Lustre: Skipped 1 previous similar message [12459.708938] Lustre: lustre-MDT0001: new disk, initializing [12459.860153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12459.888850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12459.905147] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12464.010820] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12473.831863] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12480.091932] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12488.782853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12489.128584] Lustre: lustre-OST0000: new disk, initializing [12489.131165] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12489.137819] Lustre: 344816:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12489.204260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12494.855076] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12498.548720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12498.573884] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12498.748420] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12505.267062] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12509.957524] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12510.242424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12515.824181] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12516.164484] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12519.414133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12519.417324] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12519.430992] LustreError: Skipped 1 previous similar message [12519.436356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12519.477535] Lustre: Skipped 1 previous similar message [12524.600300] Lustre: server umount lustre-OST0000 complete [12531.680615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12531.684818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12531.693532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12531.697504] Lustre: Skipped 1 previous similar message [12534.754227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12534.759462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12534.787607] Lustre: Skipped 1 previous similar message [12537.058225] Lustre: server umount lustre-MDT0000 complete [12541.610033] LustreError: 344815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825035 with bad export cookie 17836335809393978495 [12541.614117] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12541.641233] LustreError: 344815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12542.132231] Lustre: server umount lustre-MDT0001 complete [12552.743625] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 16:17:23 (1786825043) [12571.916392] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [12584.394538] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [12630.859765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12631.131093] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12631.180420] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12631.274575] Lustre: lustre-MDT0000: new disk, initializing [12631.354905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12631.377502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12636.453552] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12648.045972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12648.207806] Lustre: 349578:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12648.267142] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12648.272587] Lustre: Skipped 1 previous similar message [12648.355873] Lustre: lustre-MDT0001: new disk, initializing [12648.476195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12648.527290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12648.541712] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12653.673292] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12663.580087] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12671.807885] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12682.476539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12683.090138] Lustre: lustre-OST0000: new disk, initializing [12683.099149] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12683.116659] Lustre: 351551:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12683.294237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12690.869992] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12693.091618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12693.104641] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12693.239331] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12702.918964] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12739.553665] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12739.556812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12739.560884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12739.587149] Lustre: Skipped 1 previous similar message [12744.571829] Lustre: server umount lustre-OST0000 complete [12749.793619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12749.797440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12749.814654] Lustre: Skipped 1 previous similar message [12749.827294] Lustre: Skipped 2 previous similar messages [12754.764072] Lustre: server umount lustre-MDT0000 complete [12754.912220] LustreError: 349591:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12754.941352] LustreError: 349591:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [12758.621427] LustreError: 351550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825252 with bad export cookie 17836335809393979377 [12758.624516] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12758.627516] LustreError: 351550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12758.962880] Lustre: server umount lustre-MDT0001 complete [12773.075293] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [12775.736345] Key type lgssc unregistered [12776.098955] LNet: 353725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12776.112847] LNetError: 353725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12776.161283] LNet: Removed LNI 192.168.202.151@tcp [12777.056536] Key type .llcrypt unregistered [12777.058827] Key type ._llcrypt unregistered [12880.515538] Key type ._llcrypt registered [12880.518480] Key type .llcrypt registered [12880.610922] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [12880.943368] Key type .llcrypt unregistered [12880.945189] Key type ._llcrypt unregistered [12969.527503] Key type ._llcrypt registered [12969.530413] Key type .llcrypt registered [12969.630824] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [12970.042144] Key type .llcrypt unregistered [12970.047127] Key type ._llcrypt unregistered [13003.943526] Key type ._llcrypt registered [13003.946030] Key type .llcrypt registered [13004.070614] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [13004.487288] Key type .llcrypt unregistered [13004.496865] Key type ._llcrypt unregistered [13038.247392] Key type ._llcrypt registered [13038.248709] Key type .llcrypt registered [13038.312371] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [13038.559356] Key type .llcrypt unregistered [13038.562671] Key type ._llcrypt unregistered [13124.890518] Key type ._llcrypt registered [13124.893371] Key type .llcrypt registered [13124.996485] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [13125.276378] Key type .llcrypt unregistered [13125.277927] Key type ._llcrypt unregistered [13157.837480] Key type ._llcrypt registered [13157.842311] Key type .llcrypt registered [13158.001805] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [13158.679570] Key type .llcrypt unregistered [13158.683958] Key type ._llcrypt unregistered [13195.777691] Key type ._llcrypt registered [13195.779242] Key type .llcrypt registered [13195.888151] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [13196.127430] Key type .llcrypt unregistered [13196.133292] Key type ._llcrypt unregistered [13234.088954] Key type ._llcrypt registered [13234.090892] Key type .llcrypt registered [13234.172147] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [13234.470377] Key type .llcrypt unregistered [13234.473651] Key type ._llcrypt unregistered [13267.798973] Key type ._llcrypt registered [13267.801700] Key type .llcrypt registered [13267.914442] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [13268.354534] Key type .llcrypt unregistered [13268.357460] Key type ._llcrypt unregistered [13302.275886] Key type ._llcrypt registered [13302.277208] Key type .llcrypt registered [13302.384491] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [13302.607487] Key type .llcrypt unregistered [13302.614791] Key type ._llcrypt unregistered [13365.769531] Key type ._llcrypt registered [13365.770762] Key type .llcrypt registered [13365.824862] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [13365.985355] Key type .llcrypt unregistered [13365.986554] Key type ._llcrypt unregistered [13397.910147] Key type ._llcrypt registered [13397.914671] Key type .llcrypt registered [13398.087134] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [13398.427314] Key type .llcrypt unregistered [13398.432818] Key type ._llcrypt unregistered [13440.387066] Key type ._llcrypt registered [13440.390349] Key type .llcrypt registered [13440.593634] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [13441.048433] Key type .llcrypt unregistered [13441.051489] Key type ._llcrypt unregistered [13505.527629] Key type ._llcrypt registered [13505.532468] Key type .llcrypt registered [13505.662378] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [13525.816477] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [13527.401632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13527.551946] alg: No test for adler32 (adler32-zlib) [13528.626402] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [13528.936254] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [13530.647446] Key type lgssc registered [13531.905859] Lustre: Echo OBD driver; http://www.lustre.org/ [13572.080904] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13572.107604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13573.622342] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13573.673307] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13573.812958] Lustre: tst250fs-MDT0000: new disk, initializing [13573.920912] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13573.946287] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13580.000995] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13592.094619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13592.323544] Lustre: 374255:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13592.500830] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13592.510816] Lustre: Skipped 1 previous similar message [13592.674640] Lustre: tst250fs-MDT0001: new disk, initializing [13592.821661] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13592.901454] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13592.925082] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13598.588295] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13604.453642] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13612.789717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13613.327497] Lustre: tst250fs-OST0000: new disk, initializing [13613.337432] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13613.357809] Lustre: 375887:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13613.695862] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13620.922400] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13622.853258] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13622.870984] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13623.028403] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13633.375329] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13633.556467] Lustre: tst250fs-OST0001: new disk, initializing [13633.560560] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13633.566748] Lustre: 376762:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13633.659892] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13639.729557] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13639.744986] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13639.849069] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13640.590952] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13646.839156] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13649.581693] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13649.586329] Lustre: Skipped 2 previous similar messages [13669.856443] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13669.864266] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13669.886144] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13670.888072] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13670.893483] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13670.930309] Lustre: Skipped 1 previous similar message [13675.973170] Lustre: server umount tst250fs-MDT0000 complete [13680.822174] LustreError: 374247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826174 with bad export cookie 10615471108986093699 [13680.834040] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13680.839502] LustreError: 374247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13681.123830] Lustre: tst250fs-MDT0001-lwp-OST0000: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13681.140813] LustreError: 376759:0:(ldlm_lib.c:1192:target_handle_connect()) tst250fs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13681.153147] Lustre: Skipped 1 previous similar message [13681.186514] LustreError: 376759:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [13681.349865] Lustre: server umount tst250fs-MDT0001 complete [13687.179720] Lustre: server umount tst250fs-OST0000 complete [13692.472889] Lustre: server umount tst250fs-OST0001 complete [13713.162512] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [13720.789897] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [13751.428839] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 16:37:22 (1786826242) [13753.021922] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13771.418487] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [13783.095690] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [13818.846236] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13494 sec ========= 16:38:29 (1786826309) [13820.654927] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:38:31 (1786826311) === [13824.665892] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:38:35 (1786826315) === [13868.525838] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [13871.558566] Key type lgssc unregistered [13871.794338] LNet: 385892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13871.804904] LNetError: 385892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13871.834237] LNet: Removed LNI 192.168.202.151@tcp [13872.863156] Key type .llcrypt unregistered [13872.865165] Key type ._llcrypt unregistered