[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 491278102 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002341] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.006916] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009128] LSM: Security Framework initializing [ 0.010048] Yama: becoming mindful. [ 0.011034] SELinux: Initializing. [ 0.012067] *** VALIDATE selinux *** [ 0.019451] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023702] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024151] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025092] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026113] *** VALIDATE tmpfs *** [ 0.027412] *** VALIDATE proc *** [ 0.028238] *** VALIDATE cgroup *** [ 0.029011] *** VALIDATE cgroup2 *** [ 0.031267] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034023] Spectre V2 : User space: Vulnerable [ 0.035009] Speculative Store Bypass: Vulnerable [ 0.038136] debug: unmapping init [mem 0xffffffff88a59000-0xffffffff88a60fff] [ 0.040802] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041619] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042023] ... version: 2 [ 0.043015] ... bit width: 48 [ 0.044011] ... generic registers: 4 [ 0.045011] ... value mask: 0000ffffffffffff [ 0.046008] ... max period: 00007fffffffffff [ 0.047007] ... fixed-purpose events: 3 [ 0.048012] ... event mask: 000000070000000f [ 0.049288] rcu: Hierarchical SRCU implementation. [ 0.051514] smp: Bringing up secondary CPUs ... [ 0.052555] x86: Booting SMP configuration: [ 0.053022] .... node #0, CPUs: #1 #2 #3 [ 0.056337] smp: Brought up 1 node, 4 CPUs [ 0.058012] smpboot: Max logical packages: 1 [ 0.059017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142000] node 0 deferred pages initialised in 81ms [ 0.147098] devtmpfs: initialized [ 0.148268] x86/mm: Memory block size: 128MB [ 0.151716] gcov: version magic: 0x41383552 [ 0.154125] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155064] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157268] pinctrl core: initialized pinctrl subsystem [ 0.159177] [ 0.159470] ************************************************************* [ 0.161010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163011] ** ** [ 0.165008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166007] ** ** [ 0.168008] ** This means that this kernel is built to expose internal ** [ 0.170009] ** IOMMU data structures, which may compromise security on ** [ 0.172009] ** your system. ** [ 0.174011] ** ** [ 0.176010] ** If you see this message and you are not debugging the ** [ 0.179015] ** kernel, report this immediately to your vendor! ** [ 0.181008] ** ** [ 0.183013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.186015] ************************************************************* [ 0.188601] NET: Registered protocol family 16 [ 0.191467] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.194053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.197051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.200070] cpuidle: using governor menu [ 0.201796] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.205548] PCI: Using configuration type 1 for base access [ 0.208182] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.218112] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.221017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.225082] cryptd: max_cpu_qlen set to 1000 [ 0.228216] ACPI: Added _OSI(Module Device) [ 0.229014] ACPI: Added _OSI(Processor Device) [ 0.230013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.231031] ACPI: Added _OSI(Processor Aggregator Device) [ 0.236203] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.245551] ACPI: Interpreter enabled [ 0.247082] ACPI: PM: (supports S0 S3 S4 S5) [ 0.249019] ACPI: Using IOAPIC for interrupt routing [ 0.250166] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.254447] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.265000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.268055] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.271028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.273111] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.277331] acpiphp: Slot [2] registered [ 0.279111] acpiphp: Slot [5] registered [ 0.280109] acpiphp: Slot [6] registered [ 0.282117] acpiphp: Slot [7] registered [ 0.283122] acpiphp: Slot [8] registered [ 0.285140] acpiphp: Slot [9] registered [ 0.286138] acpiphp: Slot [10] registered [ 0.288130] acpiphp: Slot [3] registered [ 0.290117] acpiphp: Slot [4] registered [ 0.291093] acpiphp: Slot [11] registered [ 0.293102] acpiphp: Slot [12] registered [ 0.294131] acpiphp: Slot [13] registered [ 0.296163] acpiphp: Slot [14] registered [ 0.297105] acpiphp: Slot [15] registered [ 0.299117] acpiphp: Slot [16] registered [ 0.300115] acpiphp: Slot [17] registered [ 0.301135] acpiphp: Slot [18] registered [ 0.303124] acpiphp: Slot [19] registered [ 0.304111] acpiphp: Slot [20] registered [ 0.305119] acpiphp: Slot [21] registered [ 0.307190] acpiphp: Slot [22] registered [ 0.308143] acpiphp: Slot [23] registered [ 0.309140] acpiphp: Slot [24] registered [ 0.311115] acpiphp: Slot [25] registered [ 0.312160] acpiphp: Slot [26] registered [ 0.314138] acpiphp: Slot [27] registered [ 0.318179] acpiphp: Slot [28] registered [ 0.319182] acpiphp: Slot [29] registered [ 0.320136] acpiphp: Slot [30] registered [ 0.322129] acpiphp: Slot [31] registered [ 0.323068] PCI host bridge to bus 0000:00 [ 0.324026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.326028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.328029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.330030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.332042] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.335033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.337188] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.340111] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.344262] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.355015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.358000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.361021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.364030] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.366019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.369676] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.371794] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.374053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.377283] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.382023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.395024] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.400020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.405881] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.414016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.420020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.439018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.448264] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.457022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.466024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.488023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.500972] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.511025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.519016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.539017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.550864] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.557016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.562018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.578023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.587548] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.592014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.599015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.616016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.628060] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.635017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.642017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.658014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.668551] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.672378] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.675095] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.678407] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.680230] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.686030] iommu: Default domain type: Passthrough [ 0.688461] SCSI subsystem initialized [ 0.689090] ACPI: bus type USB registered [ 0.691122] usbcore: registered new interface driver usbfs [ 0.693143] usbcore: registered new interface driver hub [ 0.695085] usbcore: registered new device driver usb [ 0.697200] pps_core: LinuxPPS API ver. 1 registered [ 0.699014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.703060] PTP clock support registered [ 0.705066] EDAC MC: Ver: 3.0.0 [ 0.707145] PCI: Using ACPI for IRQ routing [ 0.709596] NetLabel: Initializing [ 0.711011] NetLabel: domain hash size = 128 [ 0.712010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.714062] NetLabel: unlabeled traffic allowed by default [ 0.715133] vgaarb: loaded [ 0.717247] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.718010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.722278] clocksource: Switched to clocksource kvm-clock [ 0.827631] VFS: Disk quotas dquot_6.6.0 [ 0.829448] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.832076] *** VALIDATE ramfs *** [ 0.833645] *** VALIDATE hugetlbfs *** [ 0.835459] pnp: PnP ACPI init [ 0.837805] pnp: PnP ACPI: found 6 devices [ 0.855659] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.859526] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.861912] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.864364] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.867076] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.869679] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.873471] NET: Registered protocol family 2 [ 0.876096] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.881358] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.884821] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.890383] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.894143] TCP: Hash tables configured (established 65536 bind 65536) [ 0.897983] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.901414] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.904475] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.907441] NET: Registered protocol family 1 [ 0.909849] RPC: Registered named UNIX socket transport module. [ 0.912357] RPC: Registered udp transport module. [ 0.914174] RPC: Registered tcp transport module. [ 0.915833] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.918278] NET: Registered protocol family 44 [ 0.920102] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.922094] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.924266] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.926934] PCI: CLS 0 bytes, default 64 [ 0.929254] Unpacking initramfs... [ 2.301700] debug: unmapping init [mem 0xffffa0f67cc54000-0xffffa0f67ffbffff] [ 2.305390] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.307745] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.310679] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.791126] Initialise system trusted keyrings [ 2.792479] Key type blacklist registered [ 2.793837] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.803945] zbud: loaded [ 2.807455] *** VALIDATE nfs *** [ 2.808696] *** VALIDATE nfs4 *** [ 2.810141] pstore: using deflate compression [ 2.813070] Platform Keyring initialized [ 2.908700] NET: Registered protocol family 38 [ 2.910348] Key type asymmetric registered [ 2.911973] Asymmetric key parser 'x509' registered [ 2.914156] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.916983] io scheduler mq-deadline registered [ 2.918518] io scheduler kyber registered [ 2.920151] io scheduler bfq registered [ 2.921741] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.924198] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.926784] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.929025] ACPI: Power Button [PWRF] [ 2.934286] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.941277] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.956671] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.964660] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.978292] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.006112] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.032744] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.038832] Non-volatile memory driver v1.3 [ 3.041038] Linux agpgart interface v0.103 [ 3.070115] virtio_blk virtio1: [vda] 150008 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.073697] vda: detected capacity change from 0 to 76804096 [ 3.087211] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.091096] vdb: detected capacity change from 0 to 1073741824 [ 3.107218] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.109891] vdc: detected capacity change from 0 to 2621440000 [ 3.126053] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.128897] vdd: detected capacity change from 0 to 2621440000 [ 3.148534] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.151800] vde: detected capacity change from 0 to 4294967296 [ 3.168056] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.171031] vdf: detected capacity change from 0 to 4294967296 [ 3.179633] libphy: Fixed MDIO Bus: probed [ 3.185878] usbcore: registered new interface driver usbserial_generic [ 3.188175] usbserial: USB Serial support registered for generic [ 3.190582] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.195910] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.197812] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.199948] mousedev: PS/2 mouse device common for all mice [ 3.202674] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.204622] rtc_cmos 00:05: RTC can wake from S4 [ 3.208965] rtc_cmos 00:05: registered as rtc0 [ 3.211550] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.212435] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.216155] intel_pstate: CPU model not supported [ 3.220880] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.224410] hid: raw HID events driver (C) Jiri Kosina [ 3.227947] usbcore: registered new interface driver usbhid [ 3.230099] usbhid: USB HID core driver [ 3.232941] drop_monitor: Initializing network drop monitor service [ 3.235646] Initializing XFRM netlink socket [ 3.238731] NET: Registered protocol family 10 [ 3.241985] Segment Routing with IPv6 [ 3.243248] NET: Registered protocol family 17 [ 3.245088] mpls_gso: MPLS GSO support [ 3.250317] RAS: Correctable Errors collector initialized. [ 3.252638] AVX version of gcm_enc/dec engaged. [ 3.254517] AES CTR mode by8 optimization enabled [ 3.326659] sched_clock: Marking stable (3326637740, 0)->(4183901485, -857263745) [ 3.330978] registered taskstats version 1 [ 3.333635] Loading compiled-in X.509 certificates [ 3.335925] zswap: loaded using pool lzo/zbud [ 3.357742] Key type big_key registered [ 3.369637] Key type encrypted registered [ 3.371117] ima: No TPM chip found, activating TPM-bypass! [ 3.372662] ima: Allocated hash algorithm: sha1 [ 3.373698] ima: No architecture policies found [ 3.374710] evm: Initialising EVM extended attributes: [ 3.376122] evm: security.selinux [ 3.377267] evm: security.ima [ 3.378298] evm: security.capability [ 3.379573] evm: HMAC attrs: 0x1 [ 3.381686] rtc_cmos 00:05: setting system clock to 2026-09-15 08:44:19 UTC (1789461859) [ 3.388213] debug: unmapping init [mem 0xffffffff89a03000-0xffffffff89bfffff] [ 3.391938] debug: unmapping init [mem 0xffffffff88782000-0xffffffff88a58fff] [ 3.401131] Write protecting the kernel read-only data: 28672k [ 3.404817] debug: unmapping init [mem 0xffffffff86e03000-0xffffffff86ffffff] [ 3.407746] debug: unmapping init [mem 0xffffffff87714000-0xffffffff877fffff] [ 3.440246] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.447659] systemd[1]: Detected virtualization kvm. [ 3.449272] systemd[1]: Detected architecture x86-64. [ 3.450743] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.476253] systemd[1]: No hostname configured. [ 3.477698] systemd[1]: Set hostname to . [ 3.479286] random: systemd: uninitialized urandom read (16 bytes read) [ 3.481298] systemd[1]: Initializing machine ID from random generator. [ 3.510428] random: ln: uninitialized urandom read (6 bytes read) [ 3.604176] random: systemd: uninitialized urandom read (16 bytes read) [ 3.606705] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.610828] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.619447] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.147231] device-mapper: uevent: version 1.0.3 [ 4.149191] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.785568] random: fast init done [ 4.796789] virtio_net virtio0 ens2: renamed from eth0 [ 4.868897] scsi host0: ata_piix [ 4.899110] scsi host1: ata_piix [ 4.900890] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.903272] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.071494] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.892216] random: crng init done [ 9.893317] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.074171] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.147295] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.412645] SELinux: Disabled at runtime. [ 11.475173] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.484964] systemd[1]: Detected virtualization kvm. [ 11.486985] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.916344] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.920608] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.925352] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.931270] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.935016] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.940972] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.944626] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ 11.986528] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.372814] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.644351] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.677721] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.801710] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.810984] EDAC sbridge: Ver: 1.1.2 [ 14.509770] Key type dns_resolver registered [ 14.820081] NFS: Registering the id_resolver key type [ 14.821534] Key type id_resolver registered [ 14.823378] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg451-server login: [ 40.716201] libcfs: loading out-of-tree module taints kernel. [ 40.737285] Key type ._llcrypt registered [ 40.738548] Key type .llcrypt registered [ 40.791262] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 52.364328] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 53.391602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 53.401943] alg: No test for adler32 (adler32-zlib) [ 54.801730] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 55.546277] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 57.314249] Key type lgssc registered [ 59.278353] Lustre: Echo OBD driver; http://www.lustre.org/ [ 79.744240] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 115.535615] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 129.409935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 129.486641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 130.543031] hrtimer: interrupt took 3970563 ns [ 130.849391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 130.900458] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 131.054125] Lustre: lustre-MDT0000: new disk, initializing [ 131.172386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 131.202249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 135.436925] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 148.620924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 148.846541] Lustre: 6483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 148.887925] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 148.893601] Lustre: Skipped 1 previous similar message [ 149.000262] Lustre: lustre-MDT0001: new disk, initializing [ 149.103638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 149.133829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 149.153421] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 153.756265] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 159.424947] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 169.932854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 170.397410] Lustre: lustre-OST0000: new disk, initializing [ 170.404726] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 170.419057] Lustre: 8421:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 170.516598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 174.373954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 174.389521] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 174.501579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 176.696959] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.350270] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 192.591895] Lustre: lustre-OST0001: new disk, initializing [ 192.596836] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 192.617886] Lustre: 9493:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 192.713988] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 198.671267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 198.686768] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 198.729456] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 199.007692] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.916278] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 216.119416] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 222.405467] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing check_logdir /tmp/testlogs/ [ 227.642347] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing yml_node [ 231.318676] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 233.779254] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 236.511845] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 238.405751] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 15 04:48:13 EDT 2026 [ 258.768982] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 260.337610] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 277.474920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 277.490469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 277.509346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 280.547094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 280.547109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 280.565796] Lustre: Skipped 1 previous similar message [ 281.856460] Lustre: server umount lustre-MDT0000 complete [ 285.424713] LustreError: 6475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789462141 with bad export cookie 16535269899781202277 [ 285.425917] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 285.454428] LustreError: 6475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 285.664871] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 285.696217] Lustre: Skipped 1 previous similar message [ 285.705653] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 285.713902] Lustre: Skipped 4 previous similar messages [ 290.791859] LustreError: 6489:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 290.806937] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 290.819387] LustreError: 6489:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 290.828884] Lustre: Skipped 1 previous similar message [ 291.992972] Lustre: server umount lustre-MDT0001 complete [ 296.446301] Lustre: server umount lustre-OST0000 complete [ 300.744484] Lustre: server umount lustre-OST0001 complete [ 307.611546] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 317.690118] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 352.244268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 352.421421] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 352.445380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 352.503454] Lustre: lustre-MDT0000: new disk, initializing [ 352.562961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 352.586684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 356.697355] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 366.996421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 367.130363] Lustre: 15046:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 367.194364] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 367.198273] Lustre: Skipped 1 previous similar message [ 367.285890] Lustre: lustre-MDT0001: new disk, initializing [ 367.378917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 367.442252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 367.463073] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 371.467929] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 380.894052] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 386.175880] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 392.849163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 393.145974] Lustre: lustre-OST0000: new disk, initializing [ 393.154037] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 393.160297] Lustre: 17014:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 393.258942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 395.019852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 395.034731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 395.187155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 400.179820] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 409.863912] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 414.663404] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 414.893659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 419.465586] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 419.744169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 425.958501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 425.958936] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 425.973057] LustreError: Skipped 1 previous similar message [ 425.989504] Lustre: Skipped 1 previous similar message [ 425.991091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 426.008967] Lustre: Skipped 1 previous similar message [ 427.737421] Lustre: server umount lustre-OST0000 complete [ 434.150666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 434.163212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 434.179381] Lustre: Skipped 1 previous similar message [ 434.185415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 438.154178] Lustre: server umount lustre-MDT0000 complete [ 441.320698] LustreError: 15924:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 441.343593] LustreError: 15924:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 442.331711] LustreError: 15036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789462298 with bad export cookie 16535269899781203810 [ 442.353210] LustreError: 15036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 442.353537] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 442.646858] Lustre: server umount lustre-MDT0001 complete [ 447.892831] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 04:51:43 (1789462303) [ 464.371781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.080130] LustreError: 19890:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 465.258827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 470.201337] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 470.501916] LustreError: 19891:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 475.622112] LustreError: 19890:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 478.885374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 484.163855] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 492.071604] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 497.349257] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 499.129253] Lustre: Failing over lustre-MDT0000 [ 499.406360] Lustre: server umount lustre-MDT0000 complete [ 499.680889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 499.687607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 499.708740] Lustre: Skipped 1 previous similar message [ 499.723313] LustreError: 19891:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 516.576137] Lustre: 3618:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789462356/real 1789462356] req@ffffa0f5c7a9a680 x1876386812945280/t0(0) o400->MGC192.168.204.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789462372 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 516.609536] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 516.625535] LustreError: 19890:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 516.640801] LustreError: 19890:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 518.628913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 526.818043] LustreError: 3615:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa0f701c8f100 x1876386812948096/t0(0) o250->MGC192.168.204.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 531.186023] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 532.452505] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 532.462111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 532.516967] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 538.801431] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 542.695264] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 542.701382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 542.707665] Lustre: Skipped 1 previous similar message [ 542.709993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 542.721918] Lustre: Skipped 3 previous similar messages [ 548.396178] Lustre: server umount lustre-MDT0000 complete [ 549.858866] LustreError: 19886:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.879794] LustreError: 19886:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 552.804912] LustreError: 19870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789462408 with bad export cookie 16535269899781205224 [ 552.806751] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 552.823820] LustreError: 19870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 553.441505] Lustre: server umount lustre-MDT0001 complete [ 563.274147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.209527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 564.216285] Lustre: Skipped 2 previous similar messages [ 568.802072] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 576.567973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 580.967611] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 587.315898] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 592.662689] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 594.547775] Lustre: Failing over lustre-MDT0000 [ 594.745371] Lustre: server umount lustre-MDT0000 complete [ 597.484476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 597.495111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 597.503764] Lustre: Skipped 1 previous similar message [ 612.737621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 612.830443] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 613.070550] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 617.599458] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 618.477538] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 618.484960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 618.501050] Lustre: Skipped 1 previous similar message [ 618.520741] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 627.006578] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 633.828803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 633.837342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 633.837980] Lustre: Skipped 2 previous similar messages [ 637.484114] Lustre: server umount lustre-MDT0000 complete [ 638.946740] LustreError: 23955:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 638.964477] LustreError: 23955:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 640.595339] LustreError: 23207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789462496 with bad export cookie 16535269899781206162 [ 640.596101] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 640.601067] LustreError: 23207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 640.949909] Lustre: server umount lustre-MDT0001 complete [ 650.338459] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 04:55:05 (1789462505) [ 666.088937] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 675.021896] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 713.924481] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 725.339075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 725.679715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 725.730368] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 725.803609] Lustre: lustre-MDT0000: new disk, initializing [ 725.873084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 725.882708] Lustre: Skipped 2 previous similar messages [ 725.896995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 730.875812] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 742.253185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 742.354310] Lustre: 29598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 742.639807] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 747.997467] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 757.982163] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 764.165716] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 771.032312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 771.295615] Lustre: 31567:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 772.835934] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 772.901944] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 772.990425] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 773.016203] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 778.169614] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 787.689350] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 793.932649] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 814.052173] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 814.052682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 814.069017] LustreError: Skipped 1 previous similar message [ 814.078157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 814.091572] Lustre: Skipped 4 previous similar messages [ 814.093750] Lustre: Skipped 1 previous similar message [ 826.336125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 826.488940] Lustre: server umount lustre-OST0000 complete [ 844.256689] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 844.562356] Lustre: server umount lustre-MDT0000 complete [ 844.772662] LustreError: 29605:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 844.793550] LustreError: 29605:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 848.103334] LustreError: 31566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789462704 with bad export cookie 16535269899781206561 [ 848.107805] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 848.116152] LustreError: 31566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 848.655889] Lustre: server umount lustre-MDT0001 complete [ 860.611125] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 863.594318] Key type lgssc unregistered [ 863.974102] LNet: 33546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.993635] LNetError: 33546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.064154] LNet: Removed LNI 192.168.204.151@tcp [ 866.250149] Key type .llcrypt unregistered [ 866.253366] Key type ._llcrypt unregistered [ 891.302015] Key type ._llcrypt registered [ 891.305237] Key type .llcrypt registered [ 891.443832] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 892.688963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 892.830917] alg: No test for adler32 (adler32-zlib) [ 893.861478] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 894.117477] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 895.810386] Key type lgssc registered [ 897.077537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 904.885246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 904.921339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 906.341157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 910.890441] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 915.944815] LustreError: 34958:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 915.970181] LustreError: 34958:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 920.109325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 920.634064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 925.140911] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 933.514916] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 940.610656] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 949.046218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 949.480721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 951.580853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 956.324903] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 964.597708] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 965.877756] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 979.925579] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 05:00:34 (1789462834) [ 982.038094] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 983.998390] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 991.405667] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 05:00:46 (1789462846) [ 1001.709027] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 05:00:56 (1789462856) [ 1002.749098] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1013.010693] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1013.017266] Lustre: Skipped 2 previous similar messages [ 1015.109595] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 05:01:10 (1789462870) [ 1017.094518] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1017.098934] Lustre: Skipped 2 previous similar messages [ 1035.275970] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1035.280502] Lustre: Skipped 7 previous similar messages [ 1037.967810] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 05:01:33 (1789462893) [ 1040.816680] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1044.458404] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1053.982560] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 05:01:48 (1789462908) [ 1056.077703] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1056.083716] Lustre: Skipped 3 previous similar messages [ 1068.466927] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 05:02:03 (1789462923) [ 1083.893304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1083.910486] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1083.911879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1083.933959] Lustre: Skipped 1 previous similar message [ 1088.450205] Lustre: server umount lustre-OST0000 complete [ 1094.113883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1094.118256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1094.128944] Lustre: Skipped 2 previous similar messages [ 1094.144345] Lustre: Skipped 1 previous similar message [ 1099.055269] Lustre: server umount lustre-MDT0000 complete [ 1099.248298] LustreError: 34958:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1099.285536] LustreError: 34958:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1103.231817] LustreError: 34937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789462959 with bad export cookie 11062730077854759747 [ 1103.238078] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1103.243739] LustreError: 34937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1103.835791] Lustre: server umount lustre-MDT0001 complete [ 1116.071648] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 1119.197210] Key type lgssc unregistered [ 1119.661633] LNet: 42831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1119.673980] LNetError: 42831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1119.691477] LNet: Removed LNI 192.168.204.151@tcp [ 1120.907164] Key type .llcrypt unregistered [ 1120.910211] Key type ._llcrypt unregistered [ 1139.592531] Key type ._llcrypt registered [ 1139.596584] Key type .llcrypt registered [ 1139.720512] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 1156.469671] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 1157.559203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1157.625171] alg: No test for adler32 (adler32-zlib) [ 1158.932241] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 1159.161206] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 1160.808700] Key type lgssc registered [ 1162.163443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1199.692845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1199.724867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1201.061172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1201.091873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1201.178379] Lustre: lustre-MDT0000: new disk, initializing [ 1201.269060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1201.282397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1206.407856] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1218.138064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1218.319870] Lustre: 45980:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1218.421855] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1218.432287] Lustre: Skipped 1 previous similar message [ 1218.525298] Lustre: lustre-MDT0001: new disk, initializing [ 1218.629415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1218.693476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1218.713585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1224.299946] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1233.369795] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1238.960175] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1246.188766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1246.573182] Lustre: lustre-OST0000: new disk, initializing [ 1246.577828] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1246.590283] Lustre: 47948:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1246.712155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1253.533575] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1256.509149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1256.525617] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1256.623259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1262.925641] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1268.051821] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1268.235367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1273.355906] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1273.626780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1277.415877] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1277.420272] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1277.429119] LustreError: Skipped 1 previous similar message [ 1277.433662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1277.442029] Lustre: Skipped 1 previous similar message [ 1277.472141] Lustre: Skipped 1 previous similar message [ 1281.794877] Lustre: server umount lustre-OST0000 complete [ 1287.650383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1287.651598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1287.662679] Lustre: Skipped 1 previous similar message [ 1291.930184] Lustre: server umount lustre-MDT0000 complete [ 1295.344372] LustreError: 45971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789463151 with bad export cookie 7890771431435243864 [ 1295.351384] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1295.365900] LustreError: 45971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1295.662848] Lustre: server umount lustre-MDT0001 complete [ 1306.170528] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 05:06:01 (1789463161) [ 1313.760333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1314.576312] LustreError: 50411:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1314.612955] LustreError: 50411:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1314.742469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1319.688901] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1319.911391] LustreError: 50412:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1324.014165] LustreError: 50411:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1328.832802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1329.123783] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1329.135962] Lustre: Skipped 1 previous similar message [ 1329.282931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1334.292306] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1342.180459] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1348.293133] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1355.321335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1355.761156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1361.953728] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1369.463656] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1377.162341] LustreError: 52939:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1377.854703] Lustre: Modifying parameter general.timeout=1 in log params [ 1379.304583] Lustre: Modifying parameter general.timeout=3 in log params [ 1379.307858] Lustre: Skipped 1 previous similar message [ 1381.495505] Lustre: Modifying parameter general.timeout=6 in log params [ 1381.506886] Lustre: Skipped 2 previous similar messages [ 1385.631881] Lustre: Modifying parameter general.timeout=11 in log params [ 1385.634889] Lustre: Skipped 4 previous similar messages [ 1393.781794] Lustre: Modifying parameter general.timeout=20 in log params [ 1393.787715] Lustre: Skipped 8 previous similar messages [ 1410.078569] Lustre: Modifying parameter general.timeout=36 in log params [ 1410.084608] Lustre: Skipped 15 previous similar messages [ 1442.585162] Lustre: Modifying parameter general.timeout=72 in log params [ 1442.591639] Lustre: Skipped 35 previous similar messages [ 1508.733992] Lustre: Disabling parameter general.timeout= in log params [ 1508.738319] Lustre: Skipped 57 previous similar messages [ 1512.020770] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 05:09:27 (1789463367) [ 1515.377379] Lustre: 59886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1517.402991] Lustre: 59984:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1519.525894] Lustre: 60082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1523.547344] Lustre: 60296:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1523.561969] Lustre: 60296:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1551.280847] Lustre: 61753:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1551.288455] Lustre: 61753:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1551.311170] LustreError: 61753:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1555.546975] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 05:10:10 (1789463410) [ 1569.765960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1569.773142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1569.788088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1573.599578] Lustre: server umount lustre-MDT0000 complete [ 1576.420938] LustreError: 50412:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1576.454628] LustreError: 50412:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1577.224399] LustreError: 53434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789463433 with bad export cookie 7890771431435244746 [ 1577.232178] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1577.236206] LustreError: 53434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1577.632280] Lustre: server umount lustre-MDT0001 complete [ 1592.028820] Lustre: server umount lustre-OST0000 complete [ 1624.206812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1624.534198] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1624.877536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1629.558349] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1638.119168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1638.273693] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1638.302978] Lustre: 64289:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1638.375992] LustreError: 64299:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1643.339715] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1651.040496] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1655.666349] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1661.870778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1662.032833] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1662.390864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1662.398120] Lustre: Skipped 1 previous similar message [ 1669.404151] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1677.630430] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1679.984692] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1679.992898] Lustre: Skipped 23 previous similar messages [ 1680.075035] Lustre: 66857:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1687.521499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1687.527931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1687.545980] Lustre: Skipped 2 previous similar messages [ 1687.553028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1687.566293] Lustre: Skipped 3 previous similar messages [ 1693.469874] Lustre: server umount lustre-OST0000 complete [ 1698.274639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1698.282444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1698.290906] Lustre: Skipped 2 previous similar messages [ 1698.314864] Lustre: Skipped 5 previous similar messages [ 1703.965656] Lustre: server umount lustre-MDT0000 complete [ 1708.057717] LustreError: 64279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789463564 with bad export cookie 7890771431435247812 [ 1708.059881] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1708.071159] LustreError: 64279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1708.489626] Lustre: server umount lustre-MDT0001 complete [ 1720.373400] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 1723.852728] Key type lgssc unregistered [ 1724.200931] LNet: 68066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1724.215025] LNetError: 68066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1724.230670] LNet: Removed LNI 192.168.204.151@tcp [ 1725.357179] Key type .llcrypt unregistered [ 1725.360713] Key type ._llcrypt unregistered [ 1750.667322] Key type ._llcrypt registered [ 1750.670368] Key type .llcrypt registered [ 1750.789848] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 1751.640801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1751.657298] alg: No test for adler32 (adler32-zlib) [ 1752.738402] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 1752.950602] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 1754.625109] Key type lgssc registered [ 1755.583670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1763.161189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1763.182642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1764.711778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1769.515349] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1774.051218] LustreError: 69475:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1774.073694] LustreError: 69475:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1778.949417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1779.172962] LustreError: 69476:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1779.847124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1784.864697] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1792.560224] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1798.767129] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1805.394731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1806.029188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1812.319670] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1819.626792] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1830.380030] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1840.098800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1840.115386] Lustre: Skipped 2 previous similar messages [ 1840.118222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1840.134842] Lustre: Skipped 2 previous similar messages [ 1845.220208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1845.228741] Lustre: Skipped 1 previous similar message [ 1846.300508] Lustre: server umount lustre-MDT0000 complete [ 1850.189253] LustreError: 69457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789463706 with bad export cookie 5807517879726757553 [ 1850.201585] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1850.340215] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1850.342681] LustreError: 69472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1850.377946] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1850.581794] Lustre: server umount lustre-MDT0001 complete [ 1856.991840] Lustre: server umount lustre-OST0000 complete [ 1885.509052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1885.919032] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1885.958416] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1886.272976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1890.727184] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1899.089728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1899.164859] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1899.201722] Lustre: 74350:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1899.300155] LustreError: 74360:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1899.713598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1904.458712] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1914.834887] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1923.129922] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1930.890595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1931.155841] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1931.648482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1938.713899] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1945.873727] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1947.448816] Lustre: 76926:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1949.243494] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1953.764339] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1953.769143] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.802731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1958.370615] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1958.380274] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1958.393542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1959.667459] Lustre: server umount lustre-OST0000 complete [ 1964.002766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1964.023279] Lustre: Skipped 1 previous similar message [ 1969.122632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1969.131551] Lustre: Skipped 4 previous similar messages [ 1977.312704] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1977.487671] Lustre: server umount lustre-MDT0000 complete [ 1979.363346] LustreError: 74357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1979.380167] LustreError: 74357:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1981.117683] LustreError: 76929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789463837 with bad export cookie 5807517879726758400 [ 1981.121647] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1981.131896] LustreError: 76929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1981.510919] Lustre: server umount lustre-MDT0001 complete [ 1992.088370] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 1994.774471] Key type lgssc unregistered [ 1995.105112] LNet: 78136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1995.121353] LNetError: 78136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1996.135540] LNet: Removed LNI 192.168.204.151@tcp [ 1997.071441] Key type .llcrypt unregistered [ 1997.076317] Key type ._llcrypt unregistered [ 2020.445698] Key type ._llcrypt registered [ 2020.448885] Key type .llcrypt registered [ 2020.562241] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 2021.741115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2021.905950] alg: No test for adler32 (adler32-zlib) [ 2022.991271] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 2023.244841] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 2025.009226] Key type lgssc registered [ 2026.103079] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2033.497161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2033.532888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2035.271936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2039.412855] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2045.415119] LustreError: 79546:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2045.442305] LustreError: 79546:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2046.444850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2046.919809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2050.863845] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2059.168605] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2065.897771] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2073.389922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2073.844865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2079.603935] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2087.077597] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2096.806172] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2097.472646] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2099.167875] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2099.173406] Lustre: Skipped 1 previous similar message [ 2101.359090] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2101.361794] Lustre: Skipped 2 previous similar messages [ 2105.467390] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2105.473077] Lustre: Skipped 4 previous similar messages [ 2113.762915] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2113.774106] Lustre: Skipped 10 previous similar messages [ 2129.891844] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2129.902860] Lustre: Skipped 18 previous similar messages [ 2162.122826] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2162.125699] Lustre: Skipped 34 previous similar messages [ 2191.857878] Lustre: 87643:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2194.488494] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 05:20:49 (1789464049) [ 2196.273987] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2198.408663] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 05:20:53 (1789464053) [ 2203.107532] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2203.119868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2203.147179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2204.132525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2204.133732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2204.151253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2209.259233] LustreError: 81371:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2209.531150] Lustre: server umount lustre-OST0000 complete [ 2215.908398] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2215.918948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2215.937855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2215.944247] Lustre: Skipped 1 previous similar message [ 2219.490212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2219.495088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2219.539065] Lustre: Skipped 1 previous similar message [ 2221.068362] Lustre: server umount lustre-MDT0000 complete [ 2224.618263] LustreError: 80276:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2224.638804] LustreError: 80276:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2225.868840] LustreError: 87645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789464081 with bad export cookie 3683600152086437035 [ 2225.886670] LustreError: 87645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2225.891032] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2226.352676] Lustre: server umount lustre-MDT0001 complete [ 2238.422658] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 2241.704802] Key type lgssc unregistered [ 2242.054027] LNet: 88893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2242.061765] LNetError: 88893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2242.078209] LNet: Removed LNI 192.168.204.151@tcp [ 2242.933942] Key type .llcrypt unregistered [ 2242.936358] Key type ._llcrypt unregistered [ 2247.510645] Key type ._llcrypt registered [ 2247.514344] Key type .llcrypt registered [ 2247.659276] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2252.582523] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules [ 2253.819684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2253.844950] alg: No test for adler32 (adler32-zlib) [ 2255.016603] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 2255.268829] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 2257.008268] Key type lgssc registered [ 2258.459364] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2259.956059] LustreError: 90092:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2278.488521] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 2281.537717] LustreError: 90092:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2287.337972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2287.360339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2288.844539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2293.223175] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2298.342385] LustreError: 91107:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2298.375221] LustreError: 91107:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2303.458990] LustreError: 91108:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2303.757083] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 05:22:38 (1789464158) [ 2308.580870] LustreError: 91107:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2309.667523] LustreError: 91139:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2309.685765] LustreError: 91139:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2309.696860] LustreError: 91139:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2310.169538] Lustre: server umount lustre-MDT0000 complete [ 2325.382958] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 2328.198938] Key type lgssc unregistered [ 2328.466375] LNet: 92633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2328.470593] LNetError: 92633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2328.478257] LNet: Removed LNI 192.168.204.151@tcp [ 2329.322438] Key type .llcrypt unregistered [ 2329.328065] Key type ._llcrypt unregistered [ 2348.168464] Key type ._llcrypt registered [ 2348.171447] Key type .llcrypt registered [ 2348.296135] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 2349.461654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2349.552780] alg: No test for adler32 (adler32-zlib) [ 2350.632937] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 2350.912664] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 2352.641105] Key type lgssc registered [ 2354.014411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2362.726481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2362.756411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2364.486517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2369.533419] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2374.627282] LustreError: 93850:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.650165] LustreError: 93850:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2379.019177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2379.782256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2384.374572] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2391.460907] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2397.938465] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2405.762772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2406.469510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2414.203760] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2422.333498] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2470.485805] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 05:25:26 (1789464326) [ 2471.910774] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2473.685885] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 05:25:29 (1789464329) [ 2477.025068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2477.027842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2477.032491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2480.611515] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2480.612413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2480.619313] Lustre: Skipped 1 previous similar message [ 2480.621305] Lustre: Skipped 1 previous similar message [ 2482.729097] Lustre: server umount lustre-MDT0000 complete [ 2485.729065] LustreError: 94860:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2485.766470] LustreError: 94860:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2486.405477] LustreError: 95703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789464342 with bad export cookie 2312299283539742905 [ 2486.422622] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2486.431045] LustreError: 95703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2486.788980] Lustre: server umount lustre-MDT0001 complete [ 2501.819340] Lustre: server umount lustre-OST0000 complete [ 2517.913299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2518.324665] LustreError: 98144:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2518.486972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2522.674475] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2523.618919] LustreError: 98145:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2528.746955] LustreError: 98144:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2530.434775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2531.117468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2535.617882] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2542.739267] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2547.860848] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2564.506061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2564.707590] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2564.711741] LustreError: 98132:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2564.731508] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2564.735878] LustreError: 100230:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2564.739273] LustreError: 100230:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2564.740926] LustreError: 100230:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2564.796447] Lustre: server umount lustre-OST0000 complete [ 2564.798093] LustreError: 100230:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2577.359824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2577.689688] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2577.701409] LustreError: 98135:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2577.737878] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2577.750412] LustreError: 100695:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2577.762441] LustreError: 100695:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2577.790778] LustreError: 100695:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2577.934801] Lustre: server umount lustre-OST0000 complete [ 2577.941275] LustreError: 100695:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2591.743952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2591.898992] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2591.903576] Lustre: Found index 0 for lustre-OST0000, updating log [ 2591.909745] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2591.971507] Lustre: lustre-OST0000: new disk, initializing [ 2591.976276] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2591.981848] Lustre: 101230:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2592.341622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2598.881751] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2603.928268] Lustre: Failing over lustre-OST0000 [ 2604.353586] Lustre: server umount lustre-OST0000 complete [ 2612.197740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2612.204123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2612.210602] Lustre: Skipped 1 previous similar message [ 2612.219395] Lustre: Skipped 2 previous similar messages [ 2616.023147] Lustre: server umount lustre-MDT0000 complete [ 2617.313405] LustreError: 100149:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2617.344192] LustreError: 100149:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2620.974363] LustreError: 98885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789464477 with bad export cookie 2312299283539745649 [ 2620.979960] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2620.982518] LustreError: 98885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2631.320065] Lustre: server umount lustre-MDT0001 complete [ 2642.185612] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 05:28:17 (1789464497) [ 2659.030868] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 2669.099910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2669.608897] LustreError: 103579:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2669.723246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2674.424899] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2682.307689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2686.924734] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2689.474112] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2695.027260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2695.712035] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2702.119831] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2706.934018] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2706.979876] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2706.981544] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2710.815420] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2710.910657] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2711.017689] Lustre: lustre-OST0001: new disk, initializing [ 2711.023437] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2711.035298] Lustre: 105908:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2711.477616] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2711.487958] Lustre: Skipped 2 previous similar messages [ 2718.169880] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2718.831760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2718.852171] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2718.964702] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2726.447644] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2736.395482] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2741.354246] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.51@tcp (stopping) [ 2744.290461] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2744.300757] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2746.315726] Lustre: server umount lustre-MDT0001 complete [ 2746.464749] LustreError: 103574:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2746.489052] LustreError: 103574:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2755.458752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2755.531917] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2755.535508] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2755.583694] Lustre: 103568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2756.973938] LustreError: 107455:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2756.980249] LustreError: 107455:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2756.991217] LustreError: 103572:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2757.105897] LustreError: 107464:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.151@tcp: cfg command failed: rc = -17 [ 2757.121334] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2757.121334] [ 2761.080785] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2761.724941] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2770.927563] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 05:30:26 (1789464626) [ 2788.180221] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 2805.140388] Lustre: 109522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2827.372675] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2831.491178] Lustre: 110658:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2914.785527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2914.801556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2914.816260] Lustre: Skipped 2 previous similar messages [ 2914.833528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2914.847036] Lustre: Skipped 3 previous similar messages [ 2919.260160] Lustre: server umount lustre-MDT0000 complete [ 2920.426573] LustreError: 103576:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2920.450503] LustreError: 103576:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 2922.878116] LustreError: 106627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789464778 with bad export cookie 2312299283539746650 [ 2922.878668] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2922.894653] LustreError: 106627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2923.283351] Lustre: server umount lustre-MDT0001 complete [ 2937.471927] Lustre: server umount lustre-OST0000 complete [ 2940.896159] Lustre: 93158:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789464781/real 1789464781] req@ffffa0f701cd5f80 x1876389220678400/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789464797 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2940.920634] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2940.937707] Lustre: Skipped 3 previous similar messages [ 2941.825755] Lustre: server umount lustre-OST0001 complete [ 2947.103169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2956.899727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2966.230471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2976.820141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2996.033342] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 3007.786075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3007.835635] Lustre: lustre-MDT0000: reset Object Index mappings [ 3008.840831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3008.852359] Lustre: Skipped 1 previous similar message [ 3014.538635] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3024.814842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3024.846238] Lustre: lustre-MDT0001: reset Object Index mappings [ 3030.425429] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3032.910367] Lustre: 115764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3039.963458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3046.649083] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3049.901776] LustreError: 116119:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3049.916786] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:231 to 0x280000400:257) [ 3049.918546] LustreError: 116119:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 3052.979166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3054.649323] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3060.241105] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 3060.243092] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3061.925756] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3069.187710] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3072.491734] Lustre: 117640:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3121.193707] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 05:36:16 (1789464976) [ 3147.238373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3147.260658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3147.260667] Lustre: Skipped 4 previous similar messages [ 3147.278715] Lustre: Skipped 2 previous similar messages [ 3152.595230] Lustre: server umount lustre-MDT0000 complete [ 3155.758588] LustreError: 114597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789465011 with bad export cookie 2312299283539793501 [ 3155.762334] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3155.766364] LustreError: 114597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3165.605403] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3165.617513] Lustre: Skipped 12 previous similar messages [ 3170.274533] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3170.541693] Lustre: server umount lustre-MDT0001 complete [ 3180.479048] Lustre: server umount lustre-OST0000 complete [ 3190.448469] Lustre: server umount lustre-OST0001 complete [ 3196.560216] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 3207.218479] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 3244.695765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3245.091292] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3245.150665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3245.375896] Lustre: lustre-MDT0000: new disk, initializing [ 3245.552380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3245.560414] Lustre: Skipped 3 previous similar messages [ 3245.580848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3249.208415] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3255.865770] Lustre: server umount lustre-MDT0000 complete [ 3265.101275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3265.618746] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3265.650530] Lustre: 121912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3265.678510] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3270.706224] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3275.413532] Lustre: server umount lustre-MDT0000 complete [ 3283.908496] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 05:38:59 (1789465139) [ 3285.877807] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3287.863811] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 05:39:03 (1789465143) [ 3304.801157] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 3314.477815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3319.884676] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3330.705492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3330.916991] Lustre: 123960:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3330.996470] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3331.001567] Lustre: Skipped 1 previous similar message [ 3331.106551] Lustre: lustre-MDT0001: new disk, initializing [ 3331.324347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3331.338479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3337.149732] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3342.827033] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3351.165717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3351.509286] Lustre: lustre-OST0000: new disk, initializing [ 3351.512803] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3351.520835] Lustre: 125539:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3353.513775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3353.532577] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3353.755530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3358.913348] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3369.623928] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3369.817727] Lustre: lustre-OST0001: new disk, initializing [ 3369.822472] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3369.831389] Lustre: 126408:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3371.089473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3371.104519] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3371.266543] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3376.499799] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3386.049737] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3389.507313] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3736.080613] Lustre: 132815:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3736.088724] Lustre: 132815:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3736.096387] Lustre: 132815:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3736.105627] Lustre: 132815:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3736.113208] Lustre: 132815:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3736.123225] Lustre: 132815:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3994.027805] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:50:48 (1789465848) [ 4011.495496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4011.496495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4011.504310] Lustre: Skipped 6 previous similar messages [ 4011.517660] Lustre: Skipped 3 previous similar messages [ 4014.570897] Lustre: server umount lustre-MDT0000 complete [ 4016.616963] LustreError: 126416:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4016.632539] LustreError: 126416:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 4017.586134] LustreError: 124700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789465873 with bad export cookie 2312299283539860890 [ 4017.586967] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4017.595219] LustreError: 124700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4021.745819] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4021.750430] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4021.764909] Lustre: Skipped 1 previous similar message [ 4021.774858] Lustre: Skipped 1 previous similar message [ 4023.912208] Lustre: server umount lustre-MDT0001 complete [ 4034.197205] Lustre: server umount lustre-OST0000 complete [ 4044.045580] Lustre: server umount lustre-OST0001 complete [ 4051.425311] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 4061.258785] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 4096.060411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4096.401241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4096.461301] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4096.538540] Lustre: lustre-MDT0000: new disk, initializing [ 4096.585735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4096.589017] Lustre: Skipped 5 previous similar messages [ 4096.605262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4101.466741] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 4113.821601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4113.990717] Lustre: 135869:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4114.094803] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4114.101781] Lustre: Skipped 1 previous similar message [ 4114.243800] Lustre: lustre-MDT0001: new disk, initializing [ 4114.401865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4114.416435] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4119.793828] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 4129.689501] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4135.478748] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4142.430701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4142.803877] Lustre: lustre-OST0000: new disk, initializing [ 4142.815656] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4142.826960] Lustre: 137838:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4144.861393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4144.874318] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4145.032326] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4149.607221] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 4158.729375] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4168.155966] Lustre: lustre-MDD0000: changelog on [ 4170.855444] Lustre: lustre-MDD0001: changelog on [ 6027.265898] Lustre: lustre-MDD0001: changelog off [ 6029.501352] Lustre: 135876:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6030.342556] Lustre: lustre-MDD0000: changelog off [ 6037.486112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6037.498600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6037.504425] Lustre: Skipped 3 previous similar messages [ 6041.115727] Lustre: server umount lustre-OST0000 complete [ 6047.716218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6047.733093] Lustre: Skipped 1 previous similar message [ 6047.746607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6047.760857] Lustre: Skipped 1 previous similar message [ 6051.893207] Lustre: server umount lustre-MDT0000 complete [ 6052.833376] LustreError: 136747:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6052.853485] LustreError: 136747:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6055.694396] LustreError: 135862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789467911 with bad export cookie 2312299283540635741 [ 6055.716107] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6056.061971] Lustre: server umount lustre-MDT0001 complete [ 6067.184920] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 6070.216623] Key type lgssc unregistered [ 6070.602559] LNet: 146975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6070.609196] LNetError: 146975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6070.633565] LNet: Removed LNI 192.168.204.151@tcp [ 6071.764170] Key type .llcrypt unregistered [ 6071.767552] Key type ._llcrypt unregistered [ 6092.035097] Key type ._llcrypt registered [ 6092.037199] Key type .llcrypt registered [ 6092.189702] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 6108.997675] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 6110.502775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6110.544406] alg: No test for adler32 (adler32-zlib) [ 6111.685660] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 6111.969677] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 6113.641791] Key type lgssc registered [ 6114.880169] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6150.210220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6150.273660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6151.535119] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6151.558316] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6151.661071] Lustre: lustre-MDT0000: new disk, initializing [ 6151.781512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6151.797734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6156.799416] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6167.454660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6167.601299] Lustre: 150186:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6167.674432] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6167.680543] Lustre: Skipped 1 previous similar message [ 6167.778585] Lustre: lustre-MDT0001: new disk, initializing [ 6167.887784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6167.935692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6167.947436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6173.618653] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6184.366952] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6190.502773] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6197.542184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6198.034278] Lustre: lustre-OST0000: new disk, initializing [ 6198.041394] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6198.066310] Lustre: 152156:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6198.223984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6206.552583] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6208.107209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6208.121430] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6208.283870] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6217.469251] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6223.636630] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_module obdecho/obdecho [ 6225.762969] LustreError: 153160:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6225.772956] LustreError: 153160:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6227.746928] LustreError: 153256:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6239.714106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6239.729198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6239.757847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6244.323116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6244.324139] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6244.339602] Lustre: Skipped 1 previous similar message [ 6244.356884] Lustre: Skipped 1 previous similar message [ 6245.615713] Lustre: server umount lustre-MDT0000 complete [ 6249.492355] LustreError: 150179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789468105 with bad export cookie 6173333984281907939 [ 6249.503714] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6249.506710] LustreError: 150179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6249.862467] Lustre: server umount lustre-MDT0001 complete [ 6264.671794] Lustre: server umount lustre-OST0000 complete [ 6287.203293] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 6299.407544] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 6340.045149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6340.357615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6340.392504] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6340.484912] Lustre: lustre-MDT0000: new disk, initializing [ 6340.599660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6340.620253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6346.122576] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6358.038635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6358.264393] Lustre: 156972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6358.380699] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6358.385080] Lustre: Skipped 1 previous similar message [ 6358.489198] Lustre: lustre-MDT0001: new disk, initializing [ 6358.556343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6358.580970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6358.593170] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6364.807881] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6375.369536] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6382.285867] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6388.788368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6389.056530] Lustre: lustre-OST0000: new disk, initializing [ 6389.059395] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6389.063426] Lustre: 158939:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6389.123290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6394.493994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6394.508397] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6394.682450] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6396.354067] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6407.406965] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6412.982437] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6413.194805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6419.021752] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6419.293259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6425.580059] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6425.586741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6425.594160] LustreError: Skipped 1 previous similar message [ 6425.602893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6425.615452] Lustre: Skipped 1 previous similar message [ 6427.848241] Lustre: server umount lustre-OST0000 complete [ 6435.302249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6435.323269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6435.355374] Lustre: Skipped 1 previous similar message [ 6435.363198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6438.716311] Lustre: server umount lustre-MDT0000 complete [ 6440.932064] LustreError: 156982:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6440.964821] LustreError: 156982:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6442.762374] LustreError: 158938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789468298 with bad export cookie 6173333984281908898 [ 6442.766056] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6442.776921] LustreError: 158938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6443.455809] Lustre: server umount lustre-MDT0001 complete [ 6447.748254] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:31:42 (1789468302) [ 6449.230801] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6451.255563] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 06:31:46 (1789468306) [ 6459.934693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6460.530959] LustreError: 161355:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6460.637583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6465.209559] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6466.018708] LustreError: 161356:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6470.117367] LustreError: 161355:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6473.362161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6479.124518] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6487.184859] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6492.662505] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6499.924063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6500.375547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6500.381236] Lustre: Skipped 1 previous similar message [ 6507.532951] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6516.893254] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6530.032137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6530.042286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6530.066689] Lustre: Skipped 1 previous similar message [ 6530.075626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6530.086657] Lustre: Skipped 1 previous similar message [ 6535.818764] Lustre: server umount lustre-MDT0000 complete [ 6537.187317] LustreError: 161355:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6537.207039] LustreError: 161355:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6539.482491] LustreError: 161337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789468395 with bad export cookie 6173333984281909780 [ 6539.493160] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6539.495183] LustreError: 161337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6542.308123] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6542.331520] Lustre: Skipped 2 previous similar messages [ 6542.340970] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6542.349301] Lustre: Skipped 4 previous similar messages [ 6546.117091] Lustre: server umount lustre-MDT0001 complete [ 6554.698555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6567.847057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6568.088565] LustreError: 165212:0:(llog.c:1655:llog_backup()) MGC192.168.204.151@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6568.101325] Lustre: 165212:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.151@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6568.932193] LustreError: 165217:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6568.951127] LustreError: 165217:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6589.409652] LustreError: 165218:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6589.443295] LustreError: 165218:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [ 6624.849896] LustreError: 165219:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6624.865886] LustreError: 165219:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 26 previous similar messages [ 6628.832627] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6628.840914] Lustre: 165212:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.151@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6669.812319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6674.924335] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6675.180372] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6683.842631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6689.095283] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6689.769523] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6695.959186] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6697.299477] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6702.672762] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6704.485763] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6728.679748] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6728.681200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6728.700665] Lustre: Skipped 2 previous similar messages [ 6728.710529] Lustre: Skipped 4 previous similar messages [ 6734.850948] Lustre: server umount lustre-MDT0000 complete [ 6738.916123] LustreError: 165219:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6738.932337] LustreError: 165219:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 43 previous similar messages [ 6739.382553] LustreError: 162092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789468595 with bad export cookie 6173333984281916234 [ 6739.402261] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6739.818243] Lustre: server umount lustre-MDT0001 complete [ 6753.876789] Lustre: server umount lustre-OST0000 complete [ 6777.063252] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 6789.450742] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 6829.770533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6829.991852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6830.014308] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6830.079735] Lustre: lustre-MDT0000: new disk, initializing [ 6830.150428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6830.153809] Lustre: Skipped 1 previous similar message [ 6830.179602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6835.799351] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6847.327704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6847.476792] Lustre: 170317:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6847.754278] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6852.882727] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6862.604593] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6869.649638] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6878.353290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6878.657779] Lustre: lustre-OST0000: new disk, initializing [ 6878.661194] Lustre: Skipped 1 previous similar message [ 6878.664669] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6878.671258] Lustre: Skipped 2 previous similar messages [ 6878.679749] Lustre: 172287:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6885.507537] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6886.085367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6886.109829] Lustre: Skipped 1 previous similar message [ 6886.122524] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6886.231623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6896.354994] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6902.029480] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6902.270635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6909.017705] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6909.302826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6915.042248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6915.064164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6915.074440] Lustre: Skipped 4 previous similar messages [ 6918.330638] Lustre: server umount lustre-OST0000 complete [ 6924.769238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6930.206955] Lustre: server umount lustre-MDT0000 complete [ 6930.402703] LustreError: 171192:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6930.410420] LustreError: 171192:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6934.722519] LustreError: 172286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789468790 with bad export cookie 6173333984281933062 [ 6934.732252] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6934.744726] LustreError: 172286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6935.161759] Lustre: server umount lustre-MDT0001 complete [ 6943.481975] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 06:39:58 (1789468798) [ 6950.820139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6956.767223] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6966.389889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6972.150238] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6980.115764] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6986.080253] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6993.665188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7003.326529] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7012.996886] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7024.489627] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 06:41:19 (1789468879) [ 7032.292522] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7032.296155] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7032.300258] LustreError: Skipped 1 previous similar message [ 7032.312070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7032.326184] Lustre: Skipped 4 previous similar messages [ 7032.356871] Lustre: Skipped 4 previous similar messages [ 7034.899587] Lustre: server umount lustre-OST0000 complete [ 7046.125763] Lustre: server umount lustre-MDT0000 complete [ 7050.986154] LustreError: 174678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789468907 with bad export cookie 6173333984281933944 [ 7050.990615] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7051.001187] LustreError: 174678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7051.938778] Lustre: server umount lustre-MDT0001 complete [ 7065.412492] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 7069.169647] Key type lgssc unregistered [ 7069.486257] LNet: 178427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7069.501709] LNetError: 178427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7069.525041] LNet: Removed LNI 192.168.204.151@tcp [ 7070.686259] Key type .llcrypt unregistered [ 7070.689632] Key type ._llcrypt unregistered [ 7091.168533] Key type ._llcrypt registered [ 7091.171928] Key type .llcrypt registered [ 7091.295540] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 7092.539491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7092.696914] alg: No test for adler32 (adler32-zlib) [ 7093.805722] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 7094.086305] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 7095.840422] Key type lgssc registered [ 7097.015661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7105.591595] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7105.626560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7121.313614] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7126.432352] LustreError: 179669:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.151@tcp: failed processing log, type 4: rc = -110 [ 7131.552556] LustreError: 179669:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7131.561969] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7131.564605] LustreError: MGC192.168.204.151@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7131.586481] LustreError: 179669:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7131.596939] LustreError: 179669:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 7131.607880] LustreError: 179669:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 7131.713451] Lustre: server umount lustre-OST0000 complete [ 7131.722408] LustreError: 179669:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7142.403467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7143.793430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7149.540064] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7153.126594] LustreError: 180171:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7153.160501] LustreError: 180171:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7158.185366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7158.249631] LustreError: 180172:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7158.467811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7163.035962] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7170.893096] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7177.873520] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7184.126438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7184.290795] LustreError: 181988:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7184.303755] Lustre: 181988:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.151@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7184.463453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7190.589146] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7198.904385] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7203.116778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7203.117863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7203.137676] Lustre: Skipped 2 previous similar messages [ 7203.146361] Lustre: Skipped 2 previous similar messages [ 7207.366992] Lustre: server umount lustre-MDT0000 complete [ 7208.418587] LustreError: 180172:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7208.441324] LustreError: 180172:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7211.290436] LustreError: 180153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789469067 with bad export cookie 3937605075844877654 [ 7211.313978] LustreError: 180153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7211.314698] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7211.364564] LustreError: 180900:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000afc89a26 x1876394193743744/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7211.706182] Lustre: server umount lustre-MDT0001 complete [ 7226.335612] Lustre: server umount lustre-OST0000 complete [ 7238.439919] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 7254.150668] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 7302.987357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7303.370461] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7303.410634] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7303.467601] Lustre: lustre-MDT0000: new disk, initializing [ 7303.536450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7303.563587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7309.413796] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7322.332962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7322.550634] Lustre: 185535:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7322.625738] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7322.629975] Lustre: Skipped 1 previous similar message [ 7322.724554] Lustre: lustre-MDT0001: new disk, initializing [ 7322.829703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7322.882872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7322.891821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7328.038502] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7338.407876] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7345.528444] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7353.351458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7353.719456] Lustre: lustre-OST0000: new disk, initializing [ 7353.725183] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7353.731719] Lustre: 187504:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7353.836165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7360.844922] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7362.153722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7362.165149] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7362.295620] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7370.561460] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7375.300510] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7375.513763] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7381.118278] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7381.280725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7388.131073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7388.132320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7388.140564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7388.171319] Lustre: Skipped 1 previous similar message [ 7389.958762] Lustre: server umount lustre-OST0000 complete [ 7394.785679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7394.797656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7394.807873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7394.816527] Lustre: Skipped 1 previous similar message [ 7398.370639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7398.375856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7400.631643] Lustre: server umount lustre-MDT0000 complete [ 7403.491220] LustreError: 186734:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7403.523897] LustreError: 186734:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7405.371432] LustreError: 185526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789469261 with bad export cookie 3937605075844878375 [ 7405.374865] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7405.384558] LustreError: 185526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7405.700388] Lustre: server umount lustre-MDT0001 complete [ 7416.417809] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 06:47:51 (1789469271) [ 7439.227158] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 7441.770246] Key type lgssc unregistered [ 7442.143027] LNet: 190448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7442.148496] LNetError: 190448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7442.181250] LNet: Removed LNI 192.168.204.151@tcp [ 7443.582242] Key type .llcrypt unregistered [ 7443.595437] Key type ._llcrypt unregistered [ 7468.249174] Key type ._llcrypt registered [ 7468.254073] Key type .llcrypt registered [ 7468.383646] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 7469.355870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7469.389964] alg: No test for adler32 (adler32-zlib) [ 7470.792908] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 7471.111298] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 7472.816235] Key type lgssc registered [ 7474.821225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7484.832840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7484.853798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7486.657835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7492.209973] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7496.170905] LustreError: 191664:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7496.209958] LustreError: 191664:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7501.290606] LustreError: 191665:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7501.462498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7501.815038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7507.103711] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7515.659502] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7522.039445] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7531.205548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7531.418242] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7531.422238] LustreError: 193528:0:(llog.c:1696:llog_backup()) MGC192.168.204.151@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7531.429886] Lustre: 193528:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.151@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7531.594870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7539.952583] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7550.010903] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7557.093618] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7557.106208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7557.130528] Lustre: Skipped 1 previous similar message [ 7557.147176] Lustre: Skipped 2 previous similar messages [ 7559.790982] Lustre: server umount lustre-MDT0000 complete [ 7562.218816] LustreError: 191659:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7562.239707] LustreError: 191659:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7564.648339] LustreError: 191643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789469420 with bad export cookie 2402140640155768466 [ 7564.656217] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7564.662479] LustreError: 191643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7565.036089] Lustre: server umount lustre-MDT0001 complete [ 7579.735917] Lustre: server umount lustre-OST0000 complete [ 7591.050400] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 7603.720547] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 7650.226954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7650.524196] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7650.562721] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7650.693871] Lustre: lustre-MDT0000: new disk, initializing [ 7650.846126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7650.907271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7656.243381] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7668.703754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7668.867943] Lustre: 197058:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7668.900456] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7668.903226] Lustre: Skipped 1 previous similar message [ 7668.905875] LustreError: 197839:0:(llog.c:1696:llog_backup()) MGC192.168.204.151@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7668.911528] LustreError: 197839:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7668.915565] Lustre: 197839:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.151@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7668.930358] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7668.933683] Lustre: Skipped 1 previous similar message [ 7669.015878] Lustre: lustre-MDT0001: new disk, initializing [ 7669.161593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7669.205478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7669.213575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7674.457994] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7684.314149] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7691.197260] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7699.986477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7700.345796] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7700.350805] Lustre: Skipped 1 previous similar message [ 7700.354434] LustreError: 199003:0:(llog.c:1696:llog_backup()) MGC192.168.204.151@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7700.362130] LustreError: 199003:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7700.371937] Lustre: 199003:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.151@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7700.385475] Lustre: 199003:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7700.442558] Lustre: lustre-OST0000: new disk, initializing [ 7700.456636] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7700.464298] Lustre: 199026:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7700.610179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7709.320578] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7709.867627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7709.889590] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7710.111264] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7720.141117] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7726.038235] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7726.371819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7731.646418] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7731.937882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7735.777787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7735.788465] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7735.793484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7735.801627] Lustre: Skipped 2 previous similar messages [ 7735.827053] Lustre: Skipped 1 previous similar message [ 7740.269781] Lustre: server umount lustre-OST0000 complete [ 7746.018981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7746.043457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7746.044161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7746.044213] LustreError: Skipped 1 previous similar message [ 7746.046336] Lustre: Skipped 1 previous similar message [ 7746.082210] Lustre: Skipped 1 previous similar message [ 7751.142818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7751.155116] Lustre: Skipped 1 previous similar message [ 7756.269475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7756.278848] Lustre: Skipped 1 previous similar message [ 7759.328664] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7759.593811] Lustre: server umount lustre-MDT0000 complete [ 7761.388390] LustreError: 197067:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7761.406435] LustreError: 197067:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7764.364586] LustreError: 197852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789469620 with bad export cookie 2402140640155769187 [ 7764.374045] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7764.374772] LustreError: 197852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7764.870798] Lustre: server umount lustre-MDT0001 complete [ 7776.058980] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 06:53:50 (1789469630) [ 7796.736303] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 7809.549764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7810.211180] LustreError: 202155:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7810.287419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7816.674225] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7820.771843] LustreError: 202156:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7820.818737] LustreError: 202156:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7827.660338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7834.005548] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7838.219847] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7845.927560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7846.316486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7846.332857] Lustre: Skipped 1 previous similar message [ 7853.634112] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7864.122773] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7864.339607] Lustre: lustre-OST0001: new disk, initializing [ 7864.342987] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7864.350469] Lustre: 204463:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7872.143817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7872.154396] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7872.345443] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7873.756756] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7888.801445] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7898.831203] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7901.519929] Lustre: DEBUG MARKER: ADD OST3 [ 7909.174712] loop: module loaded [ 7909.301936] loop0: detected capacity change from 0 to 204800000 [ 7909.698633] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7909.717157] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7909.729538] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7917.941746] Lustre: DEBUG MARKER: STOP OST3 [ 7918.322779] loop0: detected capacity change from 0 to 204800000 [ 7926.288248] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7926.590424] Lustre: lustre-OST0003: new disk, initializing [ 7926.597382] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7926.606387] Lustre: 206525:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7926.845815] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7926.857335] Lustre: Skipped 1 previous similar message [ 7928.592179] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7928.782863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7928.795646] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7928.853258] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7929.631203] Lustre: Failing over lustre-OST0003 [ 7929.958769] Lustre: server umount lustre-OST0003 complete [ 7930.657109] LustreError: 206670:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7933.049647] LustreError: 203640:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7933.080626] LustreError: 203640:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7933.959828] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7933.974410] Lustre: Skipped 1 previous similar message [ 7933.978756] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7934.971971] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7949.035760] Lustre: DEBUG MARKER: START OST3 again [ 7960.105852] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7960.722695] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7962.483717] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7962.590539] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7962.590710] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7968.364392] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7993.078059] Lustre: Failing over lustre-OST0003 [ 7993.291948] Lustre: server umount lustre-OST0003 complete [ 7993.315684] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7993.322922] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7993.351091] LustreError: 203643:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7993.376393] LustreError: 203643:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 18 previous similar messages [ 8017.384707] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8017.388494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8017.389352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8017.389360] LustreError: Skipped 1 previous similar message [ 8017.400538] Lustre: Skipped 3 previous similar messages [ 8017.407540] Lustre: Skipped 1 previous similar message [ 8019.196783] Lustre: server umount lustre-MDT0000 complete [ 8023.667505] LustreError: 202135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789469879 with bad export cookie 2402140640155770069 [ 8023.669186] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8023.681671] LustreError: 202135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8030.422199] Lustre: server umount lustre-MDT0001 complete [ 8035.905616] Lustre: server umount lustre-OST0000 complete [ 8040.806389] Lustre: server umount lustre-OST0001 complete [ 8049.065200] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 8058.764821] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 8101.998369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8102.309918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8102.336330] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8102.406903] Lustre: lustre-MDT0000: new disk, initializing [ 8102.483510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8102.485809] Lustre: Skipped 1 previous similar message [ 8102.525311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8107.661759] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8119.671584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8119.856658] Lustre: 212084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8119.936935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8119.942318] Lustre: Skipped 1 previous similar message [ 8120.063396] Lustre: lustre-MDT0001: new disk, initializing [ 8120.257437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8120.287135] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8125.723358] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8135.991961] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8142.508589] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8151.416170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8151.643421] Lustre: 214056:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8153.766660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8153.783757] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8153.960315] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8159.483489] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8171.004437] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8175.872231] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8176.062790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8181.182596] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8181.497459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8184.805964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8184.809557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8184.831265] Lustre: Skipped 4 previous similar messages [ 8184.844431] Lustre: Skipped 7 previous similar messages [ 8198.117575] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8198.397229] Lustre: server umount lustre-OST0000 complete [ 8209.922869] Lustre: server umount lustre-MDT0000 complete [ 8210.406957] LustreError: 212093:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8210.450948] LustreError: 212093:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [ 8214.501720] LustreError: 212890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789470070 with bad export cookie 2402140640155772365 [ 8214.504126] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8214.513414] LustreError: 212890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8214.912910] Lustre: server umount lustre-MDT0001 complete [ 8219.331751] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 07:01:14 (1789470074) [ 8238.997260] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 8250.916125] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 8291.676292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8291.991830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8292.030157] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8292.040439] Lustre: Skipped 1 previous similar message [ 8292.123333] Lustre: lustre-MDT0000: new disk, initializing [ 8292.126342] Lustre: Skipped 1 previous similar message [ 8292.217322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8297.398273] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8310.753316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8310.940933] Lustre: 218679:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8311.357653] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8316.502154] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8325.464668] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8333.076778] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8345.246829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8345.607983] Lustre: 220654:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8346.995526] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8347.169400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8353.587622] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8365.001107] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8371.054568] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8371.326366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8377.573336] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8377.873635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8382.958025] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8382.959145] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8382.977111] Lustre: Skipped 2 previous similar messages [ 8382.982943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8382.992405] Lustre: Skipped 7 previous similar messages [ 8386.388187] Lustre: server umount lustre-OST0000 complete [ 8397.151404] Lustre: server umount lustre-MDT0000 complete [ 8398.313475] LustreError: 220015:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8398.343965] LustreError: 220015:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8401.843163] LustreError: 218673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789470257 with bad export cookie 2402140640155773247 [ 8401.852582] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8401.853827] LustreError: 218673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8402.410266] Lustre: server umount lustre-MDT0001 complete [ 8412.250686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8412.930085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8412.944031] Lustre: Skipped 5 previous similar messages [ 8418.592784] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8430.465198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8437.361175] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8445.701646] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8452.876986] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8460.697501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8467.888823] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8476.044976] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8492.001639] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8492.008298] LustreError: Skipped 2 previous similar messages [ 8492.013201] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8492.019752] Lustre: Skipped 2 previous similar messages [ 8492.023197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8492.029453] Lustre: Skipped 2 previous similar messages [ 8498.234863] Lustre: server umount lustre-OST0000 complete [ 8508.726520] Lustre: server umount lustre-MDT0000 complete [ 8513.591421] LustreError: 222852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789470369 with bad export cookie 2402140640155774129 [ 8513.599258] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8513.599814] LustreError: 222852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8514.079379] Lustre: server umount lustre-MDT0001 complete [ 8617.929394] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 8621.076954] Key type lgssc unregistered [ 8621.415828] LNet: 226556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8621.422581] LNetError: 226556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8621.434314] LNet: Removed LNI 192.168.204.151@tcp [ 8622.427602] Key type .llcrypt unregistered [ 8622.431410] Key type ._llcrypt unregistered [ 8651.664295] Key type ._llcrypt registered [ 8651.667086] Key type .llcrypt registered [ 8651.855636] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 8668.914780] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 8669.920213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8669.955202] alg: No test for adler32 (adler32-zlib) [ 8671.190906] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 8671.478672] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 8673.144247] Key type lgssc registered [ 8674.478674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8710.883685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8710.910295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8712.274432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8712.302512] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8712.383905] Lustre: lustre-MDT0000: new disk, initializing [ 8712.573542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8712.594626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8717.864597] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8728.581831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8728.718639] Lustre: 229911:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8728.851535] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8728.859215] Lustre: Skipped 1 previous similar message [ 8728.955109] Lustre: lustre-MDT0001: new disk, initializing [ 8729.049482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8729.095430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8729.114685] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8733.852573] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8743.396920] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8748.433448] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8754.332223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8754.595629] Lustre: lustre-OST0000: new disk, initializing [ 8754.599679] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8754.605945] Lustre: 231876:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8754.695738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8761.662572] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8765.027625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8765.040125] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8765.174876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8771.827598] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8776.939779] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8777.140989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8782.385315] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8782.639554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8785.895598] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8785.900041] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8785.915446] LustreError: Skipped 1 previous similar message [ 8785.935571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8785.943587] Lustre: Skipped 1 previous similar message [ 8791.009815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8791.030427] Lustre: Skipped 1 previous similar message [ 8796.147468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8796.161160] Lustre: Skipped 1 previous similar message [ 8799.200200] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8799.531713] Lustre: server umount lustre-OST0000 complete [ 8805.857370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8805.862648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8805.871088] Lustre: Skipped 1 previous similar message [ 8805.876399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8809.815846] Lustre: server umount lustre-MDT0000 complete [ 8813.809795] LustreError: 229901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789470669 with bad export cookie 6459879425975730735 [ 8813.826883] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8814.213382] Lustre: server umount lustre-MDT0001 complete [ 8833.876244] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 8843.656331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8844.050637] LustreError: 234812:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8844.072102] LustreError: 234812:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8844.118563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8848.146277] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8849.385285] LustreError: 234813:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8854.501591] LustreError: 234813:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8856.807227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8857.203677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8862.454570] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8865.555520] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8871.925878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8872.323859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8878.959265] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8887.531500] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8887.734828] Lustre: lustre-OST0001: new disk, initializing [ 8887.738676] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8887.748358] Lustre: 237120:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8893.417435] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8895.603343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8895.613485] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8895.696850] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8903.737261] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8910.782564] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8924.549543] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 07:12:59 (1789470779) [ 8928.763860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8928.782819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8928.802067] Lustre: Skipped 1 previous similar message [ 8928.821806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8928.834405] Lustre: Skipped 2 previous similar messages [ 8931.826930] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8931.835233] Lustre: Skipped 2 previous similar messages [ 8933.212268] Lustre: server umount lustre-MDT0000 complete [ 8936.900888] LustreError: 237119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789470793 with bad export cookie 6459879425975731617 [ 8936.902354] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8936.905957] LustreError: 237119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8936.938209] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8936.944819] LustreError: 234809:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8936.960614] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8936.961880] LustreError: 234809:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8936.975258] Lustre: Skipped 5 previous similar messages [ 8937.257872] Lustre: server umount lustre-MDT0001 complete [ 8943.804237] Lustre: server umount lustre-OST0000 complete [ 8948.106116] Lustre: server umount lustre-OST0001 complete [ 8955.478401] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 8966.815683] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 9006.420658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9006.766838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9006.811404] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9006.947825] Lustre: lustre-MDT0000: new disk, initializing [ 9007.127140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9007.135293] Lustre: Skipped 1 previous similar message [ 9007.155560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9012.035336] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9024.013561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9024.189865] Lustre: 241066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9024.292832] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9024.296601] Lustre: Skipped 1 previous similar message [ 9024.456599] Lustre: lustre-MDT0001: new disk, initializing [ 9024.656692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9024.675625] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9029.995439] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9040.632976] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9046.784935] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9053.583276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9053.905321] Lustre: lustre-OST0000: new disk, initializing [ 9053.908312] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9053.918136] Lustre: 243035:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9054.040075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9054.053712] Lustre: Skipped 1 previous similar message [ 9055.804760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9055.817237] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9055.923751] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9059.464523] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9068.660983] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9073.451192] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9073.673845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9078.326475] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9078.547131] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9081.825649] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9081.827920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9081.833302] LustreError: Skipped 1 previous similar message [ 9081.836530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9081.856088] Lustre: Skipped 2 previous similar messages [ 9081.883017] Lustre: Skipped 1 previous similar message [ 9086.479636] Lustre: server umount lustre-OST0000 complete [ 9091.041656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9096.588879] Lustre: server umount lustre-MDT0000 complete [ 9097.193511] LustreError: 241078:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9097.213075] LustreError: 241078:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9100.789353] LustreError: 241057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789470956 with bad export cookie 6459879425975733745 [ 9100.793260] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9100.800894] LustreError: 241057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9101.255923] Lustre: server umount lustre-MDT0001 complete [ 9110.838657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9111.515353] LustreError: 245249:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9116.801265] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9125.996037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9126.402339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9126.407092] Lustre: Skipped 1 previous similar message [ 9130.759283] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9138.068860] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9144.861745] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9152.473438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9161.004931] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9170.416775] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9319.911907] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9319.916805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9319.936512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9319.936522] Lustre: Skipped 3 previous similar messages [ 9319.975250] Lustre: Skipped 3 previous similar messages [ 9324.321820] Lustre: server umount lustre-OST0000 complete [ 9335.267946] LustreError: 245250:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9335.287974] LustreError: 245250:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9335.377374] Lustre: server umount lustre-MDT0000 complete [ 9338.748067] LustreError: 245229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789471194 with bad export cookie 6459879425975734634 [ 9338.766988] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9338.772364] LustreError: 245229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9339.191518] Lustre: server umount lustre-MDT0001 complete [ 9441.025454] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 9444.094885] Key type lgssc unregistered [ 9444.520046] LNet: 248865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9444.528779] LNetError: 248865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9444.550402] LNet: Removed LNI 192.168.204.151@tcp [ 9445.614350] Key type .llcrypt unregistered [ 9445.615995] Key type ._llcrypt unregistered [ 9473.265024] Key type ._llcrypt registered [ 9473.269203] Key type .llcrypt registered [ 9473.374928] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 9488.448426] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 9489.612623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9489.762171] alg: No test for adler32 (adler32-zlib) [ 9490.841382] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 9491.053130] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 9492.768383] Key type lgssc registered [ 9493.969142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9544.118409] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 9554.689169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9554.720536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9556.089724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9556.122084] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9556.226418] Lustre: lustre-MDT0000: new disk, initializing [ 9556.309943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9556.333418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9561.682869] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9572.826537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9572.946855] Lustre: 252948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9573.003841] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9573.011022] Lustre: Skipped 1 previous similar message [ 9573.039890] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9573.085343] Lustre: lustre-MDT0001: new disk, initializing [ 9573.203706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9573.250032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9573.268949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9578.019543] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9582.806373] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9590.215843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9590.664281] Lustre: lustre-OST0000: new disk, initializing [ 9590.673295] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9590.680354] Lustre: 254581:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9590.810902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9596.997480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9597.008256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9597.137336] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9597.765426] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9610.050451] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9610.271587] Lustre: lustre-OST0001: new disk, initializing [ 9610.277874] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9610.288562] Lustre: 255453:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9610.413924] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9616.025349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9616.040353] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9616.157187] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9617.930340] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9628.571847] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9636.591395] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9650.150467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9650.155650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9650.180564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9652.198516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9652.201240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9652.266359] Lustre: Skipped 3 previous similar messages [ 9655.137833] Lustre: server umount lustre-MDT0000 complete [ 9659.573586] LustreError: 252942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789471515 with bad export cookie 12489121300764218651 [ 9659.576966] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9659.588448] LustreError: 252942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9660.179864] Lustre: server umount lustre-MDT0001 complete [ 9674.394265] Lustre: server umount lustre-OST0000 complete [ 9688.544652] Lustre: server umount lustre-OST0001 complete [ 9704.599651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9715.500844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9725.086806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9735.333073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9756.703568] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 9770.823632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9770.836616] Lustre: lustre-MDT0000: reset Object Index mappings [ 9771.291160] LustreError: 260463:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9771.308486] LustreError: 260463:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9771.407180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9776.613992] LustreError: 260464:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9778.204445] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9781.730329] LustreError: 260463:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9786.851971] LustreError: 260464:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9788.553306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9788.586482] Lustre: lustre-MDT0001: reset Object Index mappings [ 9789.294925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9794.589355] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9797.829937] Lustre: 261607:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9805.164494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9805.559632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9812.768496] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9814.778691] LustreError: 261959:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9816.832368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9823.101569] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9828.866222] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9831.018836] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9839.193094] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9842.767812] Lustre: 263477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9849.318871] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9849.321440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9849.336105] Lustre: Skipped 4 previous similar messages [ 9849.346324] Lustre: Skipped 1 previous similar message [ 9849.626382] Lustre: server umount lustre-MDT0000 complete [ 9853.241927] LustreError: 260445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789471709 with bad export cookie 12489121300764225441 [ 9853.261844] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9853.769443] Lustre: server umount lustre-MDT0001 complete [ 9868.452399] Lustre: server umount lustre-OST0000 complete [ 9870.306730] Lustre: 250439:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789471710/real 1789471710] req@ffffa0f6c7f31c00 x1876396707284096/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789471726 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9870.348271] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9870.364946] Lustre: Skipped 1 previous similar message [ 9872.493517] Lustre: server umount lustre-OST0001 complete [ 9894.040449] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 07:29:08 (1789471748) [ 9914.580663] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 9927.042451] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 9967.603387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9967.879435] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9967.956838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9968.107190] Lustre: lustre-MDT0000: new disk, initializing [ 9968.183307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9968.190568] Lustre: Skipped 1 previous similar message [ 9968.212500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9973.468976] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9985.185890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9985.322352] Lustre: 267737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9985.402389] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9985.407675] Lustre: Skipped 1 previous similar message [ 9985.495470] Lustre: lustre-MDT0001: new disk, initializing [ 9985.725206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9985.748929] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9990.829303] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10000.096179] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10005.716681] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10012.726043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10013.044189] Lustre: lustre-OST0000: new disk, initializing [10013.049242] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10013.060896] Lustre: 269703:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10014.595088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10014.609206] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10014.751545] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10019.830277] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10029.273471] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10034.453309] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10034.642696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10039.067706] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10039.313157] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10045.411366] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10045.414075] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10045.436193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10045.458400] Lustre: Skipped 3 previous similar messages [10047.676577] Lustre: server umount lustre-OST0000 complete [10052.066282] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10055.656082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10055.660659] Lustre: Skipped 1 previous similar message [10055.666039] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10055.687029] Lustre: Skipped 2 previous similar messages [10057.856504] Lustre: server umount lustre-MDT0000 complete [10060.769246] LustreError: 267748:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10060.800301] LustreError: 267748:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10061.693847] LustreError: 267729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789471917 with bad export cookie 12489121300764227534 [10061.707269] LustreError: 267729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10061.711695] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10061.968797] Lustre: server umount lustre-MDT0001 complete [10081.336893] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [10092.825293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10093.278820] LustreError: 272637:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10093.345604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10093.349395] Lustre: Skipped 2 previous similar messages [10098.726517] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10107.367314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10113.202648] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10117.757094] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10126.243580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10135.297155] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10144.463557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10144.721972] Lustre: lustre-OST0001: new disk, initializing [10144.726141] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10144.758959] Lustre: 274946:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10146.579569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10146.590613] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10146.730966] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10152.328801] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10162.302340] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10166.434403] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10171.851821] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10171.901968] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10182.627873] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10182.633287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10182.650990] Lustre: Skipped 3 previous similar messages [10182.661712] Lustre: Skipped 3 previous similar messages [10184.184578] Lustre: server umount lustre-MDT0000 complete [10187.757163] LustreError: 272633:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10187.790510] LustreError: 272633:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10188.339899] LustreError: 274945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789472044 with bad export cookie 12489121300764228416 [10188.341682] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10188.367627] LustreError: 274945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10188.755509] Lustre: server umount lustre-MDT0001 complete [10203.473984] Lustre: server umount lustre-OST0000 complete [10217.942049] Lustre: server umount lustre-OST0001 complete [10238.406183] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [10250.241456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10250.779192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10250.784431] Lustre: Skipped 3 previous similar messages [10255.559148] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10255.847781] LustreError: 278291:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10255.863983] LustreError: 278291:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10264.285384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10269.951794] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10273.427634] Lustre: 279431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10280.731922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10288.225617] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10295.808421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10297.849305] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10303.528165] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10305.992650] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10315.602140] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10319.548741] Lustre: 281307:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10325.586410] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10325.708951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10339.517982] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 07:36:34 (1789472194) [10354.660480] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10354.667272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10354.676672] Lustre: Skipped 2 previous similar messages [10354.695557] Lustre: Skipped 4 previous similar messages [10360.167426] Lustre: server umount lustre-MDT0000 complete [10364.296352] LustreError: 278274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789472220 with bad export cookie 12489121300764234807 [10364.305439] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10364.326968] LustreError: 278274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10370.932520] Lustre: server umount lustre-MDT0001 complete [10388.960653] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10389.142475] Lustre: server umount lustre-OST0000 complete [10393.166782] Lustre: server umount lustre-OST0001 complete [10399.720913] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [10410.890894] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [10468.897941] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [10481.837294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10482.120517] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10482.192109] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10482.326330] Lustre: lustre-MDT0000: new disk, initializing [10482.469779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10488.022811] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10499.864476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10500.061225] Lustre: 285836:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10500.419947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10506.316696] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10520.939614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10521.230483] Lustre: 287476:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10521.363240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10521.370423] Lustre: Skipped 5 previous similar messages [10523.243495] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10529.563756] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10541.628040] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10541.883258] Lustre: 288346:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10543.305699] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10543.322844] Lustre: Skipped 1 previous similar message [10543.486600] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10550.032910] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10559.079792] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10562.871394] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10562.884876] Lustre: Skipped 1 previous similar message [10578.349888] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 07:40:33 (1789472433) [10582.498628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10582.511985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10582.524791] Lustre: Skipped 3 previous similar messages [10582.534080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10582.537776] Lustre: Skipped 8 previous similar messages [10583.352738] Lustre: server umount lustre-MDT0000 complete [10584.554020] LustreError: 289335:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10584.586383] LustreError: 289335:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [10588.081284] LustreError: 286636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789472444 with bad export cookie 12489121300764242759 [10588.090164] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10588.105479] LustreError: 286636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10588.577098] Lustre: server umount lustre-MDT0001 complete [10603.524345] Lustre: server umount lustre-OST0000 complete [10605.364575] Lustre: 250439:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789472445/real 1789472445] req@ffffa0f6c8d52a00 x1876396707546752/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789472461 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10608.071514] Lustre: server umount lustre-OST0001 complete [10616.431628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10621.734227] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10628.736611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10632.050246] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10639.066554] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10646.275731] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10656.916618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10666.987478] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10675.449039] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10692.697771] loop1: detected capacity change from 0 to 204800000 [10693.028262] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10693.035647] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10693.042480] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10699.078890] loop1: detected capacity change from 0 to 204800000 [10705.754335] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10705.845706] LustreError: 290580:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs6315: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10705.858715] LustreError: 290580:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs6315 [10705.864511] LustreError: 290580:0:(mgs_handler.c:517:mgs_target_reg()) fs6315-MDT0001: Error checking target index: rc = -13 [10705.871460] LustreError: fs6315-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10705.880467] LustreError: 294311:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10705.889787] LustreError: 294311:0:(tgt_mount.c:2128:server_put_super()) no obd fs6315-MDT0001 [10705.900052] LustreError: 294311:0:(tgt_mount.c:116:server_deregister_mount()) fs6315-MDT0001 not registered [10705.939988] Lustre: server umount fs6315-MDT0001 complete [10705.943330] LustreError: 294311:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10715.927647] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10716.069070] Lustre: Setting parameter fs6315-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs6315-MDT0001 [10716.168457] Lustre: srv-fs6315-MDT0001: No data found on store. Initialize space. [10716.173421] Lustre: Skipped 4 previous similar messages [10716.269497] Lustre: fs6315-MDT0001: new disk, initializing [10716.273415] Lustre: Skipped 3 previous similar messages [10716.402570] Lustre: srv-fs6315-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10716.409902] Lustre: 294777:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs6315-MDT0001-mdtlov: can't create new llog: rc = -115 [10716.426338] LustreError: 294777:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs6315-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10721.682889] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10728.503292] Lustre: server umount fs6315-MDT0001 complete [10740.739781] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10742.912912] loop1: detected capacity change from 0 to 204800000 [10748.861197] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10749.475675] Lustre: 296346:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs6315-MDT0001-mdtlov: can't create new llog: rc = -115 [10749.491273] LustreError: 296346:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs6315-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10754.121901] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10756.100918] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10759.060699] Lustre: server umount fs6315-MDT0001 complete [10768.493789] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.51@tcp (stopping) [10771.437765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10771.459389] Lustre: Skipped 6 previous similar messages [10773.722364] Lustre: server umount lustre-MDT0000 complete [10782.827122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10783.008745] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10788.199283] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10788.856942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10795.356811] loop1: detected capacity change from 0 to 204800000 [10800.243584] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10800.685236] Lustre: 298792:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs6315-MDT0001-mdtlov: can't create new llog: rc = -115 [10800.694522] LustreError: 298792:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs6315-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10804.440647] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10808.812780] Lustre: server umount fs6315-MDT0001 complete [10829.291219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10834.417699] Lustre: server umount lustre-OST0000 complete [10853.347734] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10853.520948] Lustre: server umount lustre-MDT0000 complete [10855.393834] LustreError: 290590:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10855.417365] LustreError: 290590:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 105 previous similar messages [10857.030763] LustreError: 290572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789472713 with bad export cookie 12489121300764245440 [10857.040080] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10857.042032] LustreError: 290572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10857.375623] Lustre: server umount lustre-MDT0001 complete [10868.525995] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [10872.009148] Key type lgssc unregistered [10872.583028] LNet: 301136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10872.597648] LNetError: 301136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10872.615195] LNet: Removed LNI 192.168.204.151@tcp [10873.641307] Key type .llcrypt unregistered [10873.649490] Key type ._llcrypt unregistered [10895.060147] Key type ._llcrypt registered [10895.062674] Key type .llcrypt registered [10895.206774] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [10896.480340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10896.522629] alg: No test for adler32 (adler32-zlib) [10897.577603] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [10897.820797] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [10899.514562] Key type lgssc registered [10900.899401] Lustre: Echo OBD driver; http://www.lustre.org/ [10909.445677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10909.466796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10910.978703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10915.757657] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10921.460316] LustreError: 302404:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10921.491801] LustreError: 302404:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10924.590709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10925.090350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10929.877290] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10937.669479] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10943.915160] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10951.086630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10951.570850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10956.736512] LustreError: 304233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10958.032786] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10961.897548] LustreError: 304235:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10966.737346] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10967.011533] LustreError: 304233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10974.806117] LustreError: 304233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10974.828780] LustreError: 304233:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10985.047396] LustreError: 304235:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10985.075493] LustreError: 304235:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10988.359774] loop1: detected capacity change from 0 to 204800000 [10988.573951] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10988.596940] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10988.617772] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10994.210206] loop1: detected capacity change from 0 to 204800000 [11000.091462] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11000.332652] Lustre: fs31301-OST0000: new disk, initializing [11000.339703] Lustre: srv-fs31301-OST0000: No data found on store. Initialize space. [11000.347555] Lustre: 306141:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11000.443514] Lustre: fs31301-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11005.520846] LustreError: 304236:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11005.535291] LustreError: 304236:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [11006.873582] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11016.317581] Lustre: server umount fs31301-OST0000 complete [11030.414530] loop1: detected capacity change from 0 to 204800000 [11036.648323] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11037.019904] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11037.373542] Lustre: fs31301-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11041.370543] LustreError: 304236:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11041.407981] LustreError: 304236:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 20 previous similar messages [11044.948437] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11052.026335] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11054.999925] loop2: detected capacity change from 0 to 204800000 [11055.239987] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11055.250731] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11055.269079] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11061.478224] loop2: detected capacity change from 0 to 204800000 [11067.121692] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11067.215526] LustreError: fs31301-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11067.226324] LustreError: fs31301-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11067.231025] LustreError: 309184:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [11067.235362] LustreError: 309184:0:(tgt_mount.c:2128:server_put_super()) no obd fs31301-OST0001 [11067.238540] LustreError: 309184:0:(tgt_mount.c:116:server_deregister_mount()) fs31301-OST0001 not registered [11067.287557] Lustre: server umount fs31301-OST0001 complete [11067.291842] LustreError: 309184:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11078.854163] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11079.083376] Lustre: fs31301-OST0001: new disk, initializing [11079.087959] Lustre: srv-fs31301-OST0001: No data found on store. Initialize space. [11079.093260] Lustre: 309735:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11079.167181] Lustre: fs31301-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11086.053237] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11095.500297] Lustre: server umount fs31301-OST0000 complete [11106.231336] Lustre: server umount fs31301-OST0001 complete [11106.924080] LustreError: 304233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11106.954318] LustreError: 304233:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [11120.153781] loop1: detected capacity change from 0 to 204800000 [11127.108064] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11127.455561] Lustre: fs31301-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11134.788920] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11142.866888] loop2: detected capacity change from 0 to 204800000 [11149.480629] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11157.649710] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11168.630586] Lustre: server umount fs31301-OST0000 complete [11180.395669] Lustre: server umount fs31301-OST0001 complete [11188.210627] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11195.363608] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11195.370213] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11195.385746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11197.420127] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11197.440224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11201.546801] Lustre: server umount lustre-OST0000 complete [11206.625782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11206.639693] LustreError: Skipped 1 previous similar message [11206.643397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11206.652559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11206.663396] Lustre: Skipped 2 previous similar messages [11208.673538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11208.674554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11211.856728] Lustre: server umount lustre-MDT0000 complete [11216.365561] LustreError: 314910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789473072 with bad export cookie 17201762083176832586 [11216.386954] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11216.387989] LustreError: 314910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11217.120274] Lustre: server umount lustre-MDT0001 complete [11231.120141] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [11234.839217] Key type lgssc unregistered [11235.212598] LNet: 315662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11235.222430] LNetError: 315662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11235.240489] LNet: Removed LNI 192.168.204.151@tcp [11236.238242] Key type .llcrypt unregistered [11236.239930] Key type ._llcrypt unregistered [11276.074900] Key type ._llcrypt registered [11276.080240] Key type .llcrypt registered [11276.194784] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [11293.683962] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [11295.294605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11295.313271] alg: No test for adler32 (adler32-zlib) [11296.413322] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [11296.716475] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [11298.368301] Key type lgssc registered [11299.994041] Lustre: Echo OBD driver; http://www.lustre.org/ [11345.633181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11345.682645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11347.020659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11347.081737] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11347.237668] Lustre: lustre-MDT0000: new disk, initializing [11347.338162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11347.366258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11352.185955] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11364.255223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11364.479234] Lustre: 319520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11364.633592] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11364.646015] Lustre: Skipped 1 previous similar message [11364.761402] Lustre: lustre-MDT0001: new disk, initializing [11364.876259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11364.941055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11364.977516] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11370.053235] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11380.780350] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11387.689886] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11396.666111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11397.083538] Lustre: lustre-OST0000: new disk, initializing [11397.091338] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11397.101411] Lustre: 321490:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11397.232854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11403.903402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11403.931138] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11404.063919] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11405.457439] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11417.067736] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11419.516512] Lustre: Mounted lustre-client - version 2.17.58_40_gc6c2479 [11420.882273] Lustre: Unmounted lustre-client [11434.977830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11434.983510] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11434.984151] LustreError: Skipped 1 previous similar message [11434.987949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11435.001429] Lustre: Skipped 1 previous similar message [11435.021953] Lustre: Skipped 1 previous similar message [11436.756612] Lustre: server umount lustre-OST0000 complete [11441.634440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11441.645758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11441.685979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11445.233231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11445.236927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11448.153886] Lustre: server umount lustre-MDT0000 complete [11453.717896] LustreError: 319513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789473309 with bad export cookie 5275123848563660342 [11453.741090] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11453.747059] LustreError: 319513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11454.288715] Lustre: server umount lustre-MDT0001 complete [11467.001325] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [11469.985956] Key type lgssc unregistered [11478.048740] LNet: 1 local NIs in recovery (showing 1): 192.168.204.151@tcp [11478.563611] LNet: 323649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11478.580693] LNetError: 323649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11552.371434] LNet: Removed LNI 192.168.204.151@tcp [11553.716983] Key type .llcrypt unregistered [11553.723488] Key type ._llcrypt unregistered [11579.044283] Key type ._llcrypt registered [11579.048653] Key type .llcrypt registered [11579.210794] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [11596.485932] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [11597.553851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11597.656663] alg: No test for adler32 (adler32-zlib) [11598.791911] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [11599.139764] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [11600.992251] Key type lgssc registered [11602.697750] Lustre: Echo OBD driver; http://www.lustre.org/ [11635.357628] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 07:58:10 (1789473490) [11643.890180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11643.950743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11645.171159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11645.218737] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11645.270498] Lustre: lustre-MDT0000: new disk, initializing [11645.350036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11645.372655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11650.588602] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11663.529415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11663.698372] Lustre: 326871:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11663.791954] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11663.804772] Lustre: Skipped 1 previous similar message [11663.974835] Lustre: lustre-MDT0001: new disk, initializing [11664.103757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11664.140408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11664.154252] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11670.077223] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11681.731398] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11689.164439] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11696.898643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11697.260928] Lustre: lustre-OST0000: new disk, initializing [11697.266935] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11697.279959] Lustre: 328839:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11697.392360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11704.110331] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11704.923447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11704.930744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11705.084838] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11714.586073] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11736.037062] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11736.037967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11736.047567] LustreError: Skipped 1 previous similar message [11736.050709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11736.063128] Lustre: Skipped 1 previous similar message [11738.778544] Lustre: server umount lustre-OST0000 complete [11746.279380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11746.280569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11746.296154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11746.299701] Lustre: Skipped 1 previous similar message [11746.337512] Lustre: Skipped 2 previous similar messages [11749.327097] Lustre: server umount lustre-MDT0000 complete [11753.875546] LustreError: 329996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789473609 with bad export cookie 11449218128291454555 [11753.879322] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11753.889579] LustreError: 329996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11754.589628] Lustre: server umount lustre-MDT0001 complete [11767.828425] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [11771.097808] Key type lgssc unregistered [11771.466173] LNet: 330747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11771.494247] LNetError: 330747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11772.519604] LNet: Removed LNI 192.168.204.151@tcp [11773.455236] Key type .llcrypt unregistered [11773.457962] Key type ._llcrypt unregistered [11796.483883] Key type ._llcrypt registered [11796.485533] Key type .llcrypt registered [11796.632474] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [11797.444807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11797.481039] alg: No test for adler32 (adler32-zlib) [11798.838860] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [11799.157462] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [11800.802875] Key type lgssc registered [11802.111909] Lustre: Echo OBD driver; http://www.lustre.org/ [11811.214348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11811.229284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11812.709223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11818.238643] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11822.051765] LustreError: 332038:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11822.069957] LustreError: 332038:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11827.170378] LustreError: 332037:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11827.205826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11827.550369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11832.460504] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11839.759924] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11846.452970] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11854.177374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11854.708088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11862.219725] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11869.712470] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11880.342868] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11882.020886] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11887.597539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11887.600277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11887.610732] Lustre: Skipped 1 previous similar message [11888.248635] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.51@tcp (stopping) [11888.260650] Lustre: Skipped 2 previous similar messages [11892.706549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11892.712291] Lustre: Skipped 2 previous similar messages [11897.831709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11897.847751] Lustre: Skipped 3 previous similar messages [11900.385373] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11900.768253] Lustre: server umount lustre-MDT0000 complete [11902.948293] LustreError: 333774:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11902.973484] LustreError: 333774:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11905.211954] LustreError: 332018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789473761 with bad export cookie 4584482144719279014 [11905.218851] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11905.221788] LustreError: 332018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11908.066808] LustreError: 333774:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11908.068933] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11908.092348] LustreError: 333774:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11908.131885] Lustre: Skipped 1 previous similar message [11908.136958] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11908.147576] Lustre: Skipped 1 previous similar message [11911.660343] Lustre: server umount lustre-MDT0001 complete [11915.905533] Lustre: server umount lustre-OST0000 complete [11925.365298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11930.678622] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11933.453437] Lustre: 336219:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.151@tcp [11935.516483] Lustre: server umount MGS complete [11944.188230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11944.651484] LustreError: 336791:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11944.691148] LustreError: 336791:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11944.794189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11949.506191] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11953.746303] LustreError: 336787:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11953.764543] LustreError: 336787:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11958.111422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11958.717994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11962.818307] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11970.025184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11970.325407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11977.463838] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11986.504756] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11988.651417] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11994.951683] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11996.360092] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12003.376746] Lustre: server umount lustre-OST0000 complete [12009.953611] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12009.963240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12009.982381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12009.986638] Lustre: Skipped 2 previous similar messages [12017.756634] Lustre: server umount lustre-MDT0000 complete [12020.705559] Lustre: 331339:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789473860/real 1789473860] req@00000000960e32c6 x1876399127327872/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789473876 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12020.713549] LustreError: 338667:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12020.725983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12020.725992] Lustre: Skipped 1 previous similar message [12020.778055] LustreError: 338667:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12021.621574] LustreError: 336773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789473877 with bad export cookie 4584482144719279854 [12021.629064] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12021.638356] LustreError: 336773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12021.906124] Lustre: server umount lustre-MDT0001 complete [12034.194596] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [12036.885682] Key type lgssc unregistered [12037.130750] LNet: 340229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12037.135341] LNetError: 340229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12037.149789] LNet: Removed LNI 192.168.204.151@tcp [12037.851608] Key type .llcrypt unregistered [12037.861656] Key type ._llcrypt unregistered [12056.075676] Key type ._llcrypt registered [12056.078797] Key type .llcrypt registered [12056.187965] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [12072.032246] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [12073.023938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12073.041979] alg: No test for adler32 (adler32-zlib) [12074.011801] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [12074.177724] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [12075.858839] Key type lgssc registered [12077.109414] Lustre: Echo OBD driver; http://www.lustre.org/ [12112.485922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12112.559496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12113.958890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12113.993368] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12114.137650] Lustre: lustre-MDT0000: new disk, initializing [12114.297618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12114.315747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12119.150977] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [12130.875594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12131.026962] Lustre: 343386:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12131.098483] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12131.106828] Lustre: Skipped 1 previous similar message [12131.204764] Lustre: lustre-MDT0001: new disk, initializing [12131.333291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12131.415899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12131.442065] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12136.479605] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [12145.336546] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12150.359177] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12157.115380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12157.425757] Lustre: lustre-OST0000: new disk, initializing [12157.430846] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12157.446397] Lustre: 345354:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12157.590199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12163.171123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12163.176599] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12163.289153] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12163.407844] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [12173.197915] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12179.441319] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12179.939155] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12184.251869] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12184.596924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12189.157184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12189.162137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12189.191177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12189.194847] Lustre: Skipped 1 previous similar message [12192.899387] Lustre: server umount lustre-OST0000 complete [12197.857615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12197.861844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12197.873917] Lustre: Skipped 1 previous similar message [12197.881500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12199.404569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12199.418098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12202.823494] Lustre: server umount lustre-MDT0000 complete [12206.783041] LustreError: 343377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789474062 with bad export cookie 12836475037611564728 [12206.785800] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12206.789533] LustreError: 343377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12207.147140] Lustre: server umount lustre-MDT0001 complete [12216.505292] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 08:07:51 (1789474071) [12236.807884] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [12247.903408] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [12282.971955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12283.281757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12283.310995] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12283.391174] Lustre: lustre-MDT0000: new disk, initializing [12283.493283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12283.515668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12288.016662] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [12298.488739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12298.622516] Lustre: 350117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12298.668416] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12298.672344] Lustre: Skipped 1 previous similar message [12298.759555] Lustre: lustre-MDT0001: new disk, initializing [12298.872328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12298.928395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12298.934137] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12303.618640] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [12313.942532] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12319.280620] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12325.561365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12325.836238] Lustre: lustre-OST0000: new disk, initializing [12325.842291] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12325.856943] Lustre: 352081:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12325.979904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12332.607611] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [12335.651160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12335.659710] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12335.742880] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12342.580041] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12371.436914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12371.444984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12371.445593] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12371.451845] Lustre: Skipped 1 previous similar message [12371.470555] Lustre: Skipped 2 previous similar messages [12376.550111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12376.554350] Lustre: Skipped 1 previous similar message [12376.773225] Lustre: server umount lustre-OST0000 complete [12380.658972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12380.672581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12386.181525] Lustre: server umount lustre-MDT0000 complete [12386.788577] LustreError: 350124:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12386.811545] LustreError: 350124:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12390.214818] LustreError: 350108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789474246 with bad export cookie 12836475037611565610 [12390.216148] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12390.222022] LustreError: 350108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12390.684213] Lustre: server umount lustre-MDT0001 complete [12402.505785] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [12405.079867] Key type lgssc unregistered [12405.325445] LNet: 354235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12405.336433] LNetError: 354235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12405.364175] LNet: Removed LNI 192.168.204.151@tcp [12406.243953] Key type .llcrypt unregistered [12406.246191] Key type ._llcrypt unregistered [12491.663193] Key type ._llcrypt registered [12491.665223] Key type .llcrypt registered [12491.763557] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [12492.095107] Key type .llcrypt unregistered [12492.103765] Key type ._llcrypt unregistered [12563.132380] Key type ._llcrypt registered [12563.135933] Key type .llcrypt registered [12563.266727] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [12563.580233] Key type .llcrypt unregistered [12563.583410] Key type ._llcrypt unregistered [12591.238465] Key type ._llcrypt registered [12591.240759] Key type .llcrypt registered [12591.333623] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [12591.555980] Key type .llcrypt unregistered [12591.558522] Key type ._llcrypt unregistered [12619.309934] Key type ._llcrypt registered [12619.317627] Key type .llcrypt registered [12619.468410] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [12619.775504] Key type .llcrypt unregistered [12619.778949] Key type ._llcrypt unregistered [12694.446967] Key type ._llcrypt registered [12694.449635] Key type .llcrypt registered [12694.569452] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [12694.792746] Key type .llcrypt unregistered [12694.798661] Key type ._llcrypt unregistered [12724.401921] Key type ._llcrypt registered [12724.404768] Key type .llcrypt registered [12724.539748] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [12724.799199] Key type .llcrypt unregistered [12724.802450] Key type ._llcrypt unregistered [12754.160192] Key type ._llcrypt registered [12754.166463] Key type .llcrypt registered [12754.304913] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [12754.598741] Key type .llcrypt unregistered [12754.601918] Key type ._llcrypt unregistered [12782.063645] Key type ._llcrypt registered [12782.065496] Key type .llcrypt registered [12782.166916] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [12782.601617] Key type .llcrypt unregistered [12782.603221] Key type ._llcrypt unregistered [12816.053382] Key type ._llcrypt registered [12816.054933] Key type .llcrypt registered [12816.143266] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [12816.446527] Key type .llcrypt unregistered [12816.448776] Key type ._llcrypt unregistered [12847.072249] Key type ._llcrypt registered [12847.074485] Key type .llcrypt registered [12847.165974] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [12847.403442] Key type .llcrypt unregistered [12847.408764] Key type ._llcrypt unregistered [12906.307760] Key type ._llcrypt registered [12906.310665] Key type .llcrypt registered [12906.409363] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [12906.795633] Key type .llcrypt unregistered [12906.799437] Key type ._llcrypt unregistered [12932.314189] Key type ._llcrypt registered [12932.317120] Key type .llcrypt registered [12932.382572] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [12932.613659] Key type .llcrypt unregistered [12932.617763] Key type ._llcrypt unregistered [12960.803426] Key type ._llcrypt registered [12960.805098] Key type .llcrypt registered [12960.969687] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [12961.573531] Key type .llcrypt unregistered [12961.577622] Key type ._llcrypt unregistered [13010.161677] Key type ._llcrypt registered [13010.164651] Key type .llcrypt registered [13010.315364] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [13026.614923] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [13027.536372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13027.566500] alg: No test for adler32 (adler32-zlib) [13028.823810] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [13029.110562] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [13030.824235] Key type lgssc registered [13032.061221] Lustre: Echo OBD driver; http://www.lustre.org/ [13066.604148] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13066.709182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13067.971829] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13068.024318] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13068.096327] Lustre: tst250fs-MDT0000: new disk, initializing [13068.186655] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13068.215881] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13072.916490] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [13083.139297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13083.315266] Lustre: 374486:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13083.363208] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13083.366632] Lustre: Skipped 1 previous similar message [13083.458930] Lustre: tst250fs-MDT0001: new disk, initializing [13083.564925] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13083.607257] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13083.618377] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13089.372970] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [13094.815862] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13101.811749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13102.064937] Lustre: tst250fs-OST0000: new disk, initializing [13102.069377] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13102.074842] Lustre: 376117:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13102.161134] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13106.799726] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13106.808485] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13106.967115] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13108.965832] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [13119.913361] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13120.200994] Lustre: tst250fs-OST0001: new disk, initializing [13120.204217] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13120.210893] Lustre: 376988:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13120.347954] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13126.284391] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13126.298981] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13126.386460] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13127.369519] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [13133.223871] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13137.107463] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13137.114694] Lustre: Skipped 2 previous similar messages [13155.297613] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13155.304533] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13155.317972] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13157.363083] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13157.378153] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13157.411378] Lustre: Skipped 2 previous similar messages [13162.481768] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13162.493929] Lustre: Skipped 4 previous similar messages [13167.586360] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13167.593889] Lustre: Skipped 3 previous similar messages [13169.120201] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13169.561856] Lustre: server umount tst250fs-MDT0000 complete [13173.149398] LustreError: 375279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789475029 with bad export cookie 13705125958857029489 [13173.151519] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13173.166755] LustreError: 375279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13173.410273] Lustre: server umount tst250fs-MDT0001 complete [13187.199632] Lustre: server umount tst250fs-OST0000 complete [13201.524212] Lustre: server umount tst250fs-OST0001 complete [13218.924962] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [13227.673202] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [13261.717823] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 08:25:17 (1789475117) [13263.137873] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13279.450612] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [13290.116770] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [13319.578286] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13079 sec ========= 08:26:14 (1789475174) [13321.022749] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:26:16 (1789475176) === [13324.072271] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:26:19 (1789475179) === [13362.404327] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [13365.207687] Key type lgssc unregistered [13365.538201] LNet: 386138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13365.544985] LNetError: 386138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13365.560985] LNet: Removed LNI 192.168.204.151@tcp [13366.640476] Key type .llcrypt unregistered [13366.644194] Key type ._llcrypt unregistered