[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 461286018 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003365] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008448] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.011131] LSM: Security Framework initializing [ 0.012050] Yama: becoming mindful. [ 0.013036] SELinux: Initializing. [ 0.014073] *** VALIDATE selinux *** [ 0.022379] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027352] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028139] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031022] *** VALIDATE tmpfs *** [ 0.032457] *** VALIDATE proc *** [ 0.033236] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035280] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036150] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038028] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.042265] debug: unmapping init [mem 0xffffffff8ae59000-0xffffffff8ae60fff] [ 0.044175] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045711] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046023] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051013] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053290] rcu: Hierarchical SRCU implementation. [ 0.055389] smp: Bringing up secondary CPUs ... [ 0.056616] x86: Booting SMP configuration: [ 0.057040] .... node #0, CPUs: #1 #2 #3 [ 0.060469] smp: Brought up 1 node, 4 CPUs [ 0.062013] smpboot: Max logical packages: 1 [ 0.063018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138623] node 0 deferred pages initialised in 73ms [ 0.142668] devtmpfs: initialized [ 0.144214] x86/mm: Memory block size: 128MB [ 0.146498] gcov: version magic: 0x41383552 [ 0.149273] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150058] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151261] pinctrl core: initialized pinctrl subsystem [ 0.152194] [ 0.152848] ************************************************************* [ 0.153013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154015] ** ** [ 0.155025] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156013] ** ** [ 0.157012] ** This means that this kernel is built to expose internal ** [ 0.158014] ** IOMMU data structures, which may compromise security on ** [ 0.159015] ** your system. ** [ 0.160012] ** ** [ 0.161013] ** If you see this message and you are not debugging the ** [ 0.162013] ** kernel, report this immediately to your vendor! ** [ 0.163011] ** ** [ 0.164014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165012] ************************************************************* [ 0.166678] NET: Registered protocol family 16 [ 0.167460] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.168048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.169055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.170566] cpuidle: using governor menu [ 0.172757] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.175490] PCI: Using configuration type 1 for base access [ 0.177135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.188120] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.190094] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.193156] cryptd: max_cpu_qlen set to 1000 [ 0.197230] ACPI: Added _OSI(Module Device) [ 0.199014] ACPI: Added _OSI(Processor Device) [ 0.200012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.202012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.206126] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.214037] ACPI: Interpreter enabled [ 0.215043] ACPI: PM: (supports S0 S3 S4 S5) [ 0.216015] ACPI: Using IOAPIC for interrupt routing [ 0.218106] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.223587] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.239286] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.241039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.244018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.247071] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.252577] acpiphp: Slot [2] registered [ 0.255154] acpiphp: Slot [5] registered [ 0.257148] acpiphp: Slot [6] registered [ 0.259149] acpiphp: Slot [7] registered [ 0.261138] acpiphp: Slot [8] registered [ 0.263165] acpiphp: Slot [9] registered [ 0.266135] acpiphp: Slot [10] registered [ 0.268124] acpiphp: Slot [3] registered [ 0.270109] acpiphp: Slot [4] registered [ 0.272133] acpiphp: Slot [11] registered [ 0.275114] acpiphp: Slot [12] registered [ 0.276149] acpiphp: Slot [13] registered [ 0.278076] acpiphp: Slot [14] registered [ 0.280093] acpiphp: Slot [15] registered [ 0.281098] acpiphp: Slot [16] registered [ 0.283169] acpiphp: Slot [17] registered [ 0.285086] acpiphp: Slot [18] registered [ 0.287133] acpiphp: Slot [19] registered [ 0.288097] acpiphp: Slot [20] registered [ 0.290099] acpiphp: Slot [21] registered [ 0.291098] acpiphp: Slot [22] registered [ 0.292134] acpiphp: Slot [23] registered [ 0.294089] acpiphp: Slot [24] registered [ 0.295102] acpiphp: Slot [25] registered [ 0.297093] acpiphp: Slot [26] registered [ 0.298095] acpiphp: Slot [27] registered [ 0.300101] acpiphp: Slot [28] registered [ 0.301113] acpiphp: Slot [29] registered [ 0.303116] acpiphp: Slot [30] registered [ 0.304128] acpiphp: Slot [31] registered [ 0.306075] PCI host bridge to bus 0000:00 [ 0.308026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.310024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.313024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.315020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.318028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.320042] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.323272] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.327080] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.330422] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.341732] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.346899] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.349032] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.352043] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.354026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.357336] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.361950] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.364040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.367696] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.372014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.385014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.391015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.398603] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.407026] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.414014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.430041] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.440991] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.454017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.465014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.502026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.534817] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.552021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.560027] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.582025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.594252] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.604029] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.610023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.631068] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.650011] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.657024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.667020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.694034] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.708000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.719027] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.727022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.745022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.757036] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.759449] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.762418] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.765431] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.769311] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.774170] iommu: Default domain type: Passthrough [ 0.776654] SCSI subsystem initialized [ 0.778163] ACPI: bus type USB registered [ 0.779147] usbcore: registered new interface driver usbfs [ 0.781104] usbcore: registered new interface driver hub [ 0.784136] usbcore: registered new device driver usb [ 0.786210] pps_core: LinuxPPS API ver. 1 registered [ 0.788017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.791113] PTP clock support registered [ 0.793202] EDAC MC: Ver: 3.0.0 [ 0.795092] PCI: Using ACPI for IRQ routing [ 0.798407] NetLabel: Initializing [ 0.800012] NetLabel: domain hash size = 128 [ 0.803013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.806082] NetLabel: unlabeled traffic allowed by default [ 0.810088] vgaarb: loaded [ 0.812297] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.815044] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.822434] clocksource: Switched to clocksource kvm-clock [ 0.934806] VFS: Disk quotas dquot_6.6.0 [ 0.936379] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.938910] *** VALIDATE ramfs *** [ 0.940166] *** VALIDATE hugetlbfs *** [ 0.941504] pnp: PnP ACPI init [ 0.943733] pnp: PnP ACPI: found 6 devices [ 0.960869] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.964480] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.966703] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.969083] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.971914] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.976635] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.979799] NET: Registered protocol family 2 [ 0.982650] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.987876] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.991703] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.997790] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.001810] TCP: Hash tables configured (established 65536 bind 65536) [ 1.005163] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.008957] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.012358] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.015870] NET: Registered protocol family 1 [ 1.019477] RPC: Registered named UNIX socket transport module. [ 1.021886] RPC: Registered udp transport module. [ 1.023927] RPC: Registered tcp transport module. [ 1.025842] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.028381] NET: Registered protocol family 44 [ 1.030177] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.032397] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.034641] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.036987] PCI: CLS 0 bytes, default 64 [ 1.038819] Unpacking initramfs... [ 2.484398] debug: unmapping init [mem 0xffff88947cc54000-0xffff88947ffbffff] [ 2.491293] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.493793] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.498586] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.016862] Initialise system trusted keyrings [ 3.018588] Key type blacklist registered [ 3.020838] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.030916] zbud: loaded [ 3.034487] *** VALIDATE nfs *** [ 3.035886] *** VALIDATE nfs4 *** [ 3.038566] pstore: using deflate compression [ 3.042507] Platform Keyring initialized [ 3.152440] NET: Registered protocol family 38 [ 3.154222] Key type asymmetric registered [ 3.155685] Asymmetric key parser 'x509' registered [ 3.157475] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.160479] io scheduler mq-deadline registered [ 3.162159] io scheduler kyber registered [ 3.164036] io scheduler bfq registered [ 3.166104] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.169326] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.172561] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.175555] ACPI: Power Button [PWRF] [ 3.181155] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.189723] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.220568] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.228784] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.252375] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.279276] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.306464] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.311720] Non-volatile memory driver v1.3 [ 3.313416] Linux agpgart interface v0.103 [ 3.342521] virtio_blk virtio1: [vda] 146624 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.345246] vda: detected capacity change from 0 to 75071488 [ 3.358855] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.361310] vdb: detected capacity change from 0 to 1073741824 [ 3.375861] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.378283] vdc: detected capacity change from 0 to 2621440000 [ 3.393814] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.396881] vdd: detected capacity change from 0 to 2621440000 [ 3.412895] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.416200] vde: detected capacity change from 0 to 4294967296 [ 3.433456] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.436866] vdf: detected capacity change from 0 to 4294967296 [ 3.444901] libphy: Fixed MDIO Bus: probed [ 3.450588] usbcore: registered new interface driver usbserial_generic [ 3.453379] usbserial: USB Serial support registered for generic [ 3.456303] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.461942] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.463908] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.466807] mousedev: PS/2 mouse device common for all mice [ 3.469443] rtc_cmos 00:05: RTC can wake from S4 [ 3.472639] rtc_cmos 00:05: registered as rtc0 [ 3.474329] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.477204] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.480987] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.482086] intel_pstate: CPU model not supported [ 3.487789] hid: raw HID events driver (C) Jiri Kosina [ 3.488302] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.490269] usbcore: registered new interface driver usbhid [ 3.495154] usbhid: USB HID core driver [ 3.496996] drop_monitor: Initializing network drop monitor service [ 3.499865] Initializing XFRM netlink socket [ 3.502087] NET: Registered protocol family 10 [ 3.505789] Segment Routing with IPv6 [ 3.507442] NET: Registered protocol family 17 [ 3.509777] mpls_gso: MPLS GSO support [ 3.516279] RAS: Correctable Errors collector initialized. [ 3.518937] AVX version of gcm_enc/dec engaged. [ 3.520969] AES CTR mode by8 optimization enabled [ 3.606081] sched_clock: Marking stable (3606015683, 0)->(4563409520, -957393837) [ 3.609396] registered taskstats version 1 [ 3.612154] Loading compiled-in X.509 certificates [ 3.614764] zswap: loaded using pool lzo/zbud [ 3.641280] Key type big_key registered [ 3.656897] Key type encrypted registered [ 3.658894] ima: No TPM chip found, activating TPM-bypass! [ 3.661155] ima: Allocated hash algorithm: sha1 [ 3.664460] ima: No architecture policies found [ 3.666023] evm: Initialising EVM extended attributes: [ 3.668050] evm: security.selinux [ 3.669238] evm: security.ima [ 3.670108] evm: security.capability [ 3.671473] evm: HMAC attrs: 0x1 [ 3.673896] rtc_cmos 00:05: setting system clock to 2026-08-28 02:22:39 UTC (1787883759) [ 3.679861] debug: unmapping init [mem 0xffffffff8be03000-0xffffffff8bffffff] [ 3.682513] debug: unmapping init [mem 0xffffffff8ab82000-0xffffffff8ae58fff] [ 3.690075] Write protecting the kernel read-only data: 28672k [ 3.693360] debug: unmapping init [mem 0xffffffff89203000-0xffffffff893fffff] [ 3.695820] debug: unmapping init [mem 0xffffffff89b14000-0xffffffff89bfffff] [ 3.742795] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.752601] systemd[1]: Detected virtualization kvm. [ 3.754756] systemd[1]: Detected architecture x86-64. [ 3.757112] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.786446] systemd[1]: No hostname configured. [ 3.788300] systemd[1]: Set hostname to . [ 3.790351] random: systemd: uninitialized urandom read (16 bytes read) [ 3.793106] systemd[1]: Initializing machine ID from random generator. [ 3.929313] random: systemd: uninitialized urandom read (16 bytes read) [ 3.931778] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.935850] random: systemd: uninitialized urandom read (16 bytes read) [ 3.938311] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.943361] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.581057] device-mapper: uevent: version 1.0.3 [ 4.583247] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK [ 4.820557] random: fast init done ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.297687] virtio_net virtio0 ens2: renamed from eth0 [ 5.322732] scsi host0: ata_piix [ 5.354503] scsi host1: ata_piix [ 5.356459] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.359313] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.061389] dracut-initqueue[578]: RTNETLINK answers: File exists [ 10.175890] random: crng init done [ 10.177680] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.549703] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.768575] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.060662] SELinux: Disabled at runtime. [ 12.126076] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.143996] systemd[1]: Detected virtualization kvm. [ 12.145558] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.614358] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.617943] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.624201] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.630706] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.634505] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.642064] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.661118] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ 12.809466] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.111708] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.449255] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.496885] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.603572] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.616424] EDAC sbridge: Ver: 1.1.2 [ 15.068559] Key type dns_resolver registered [ 15.416078] NFS: Registering the id_resolver key type [ 15.418376] Key type id_resolver registered [ 15.421151] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg341-server login: [ 60.487104] libcfs: loading out-of-tree module taints kernel. [ 60.585648] Key type ._llcrypt registered [ 60.589507] Key type .llcrypt registered [ 60.750517] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [ 78.076191] hrtimer: interrupt took 6640568 ns [ 81.721100] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 83.565435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 83.581345] alg: No test for adler32 (adler32-zlib) [ 84.917187] Lustre: Lustre: Build Version: 2.17.57_84_gd387c42 [ 86.015584] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 87.856161] Key type lgssc registered [ 90.590031] Lustre: Echo OBD driver; http://www.lustre.org/ [ 111.735202] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 155.906161] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 169.360579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 169.398303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 170.751754] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 170.777194] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 170.868359] Lustre: lustre-MDT0000: new disk, initializing [ 170.962062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 171.005318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 175.472819] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.878421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 189.044957] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 189.090512] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 189.094078] Lustre: Skipped 1 previous similar message [ 189.211086] Lustre: lustre-MDT0001: new disk, initializing [ 189.282555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 189.336677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 189.350071] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 194.813952] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 199.915540] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 210.411415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 210.893380] Lustre: lustre-OST0000: new disk, initializing [ 210.904704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 210.920671] Lustre: 8453:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 211.004697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 217.635703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 217.646218] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 217.680039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 218.696775] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.648228] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 234.765961] Lustre: lustre-OST0001: new disk, initializing [ 234.775395] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 234.780271] Lustre: 9526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 234.850404] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 241.646041] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 242.743099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 242.767488] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 242.877406] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 254.574932] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 260.785366] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 268.582116] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing check_logdir /tmp/testlogs/ [ 275.147076] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing yml_node [ 280.652087] Lustre: DEBUG MARKER: Client: 2.17.57.84 [ 283.411930] Lustre: DEBUG MARKER: MDS: 2.17.57.84 [ 286.247864] Lustre: DEBUG MARKER: OSS: 2.17.57.84 [ 288.007354] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 22:27:21 EDT 2026 [ 307.317926] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 309.205348] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 327.649417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 327.654055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 327.676210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 330.210110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 330.212051] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.220643] Lustre: Skipped 1 previous similar message [ 330.254261] Lustre: Skipped 2 previous similar messages [ 331.643371] Lustre: server umount lustre-MDT0000 complete [ 335.629220] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787884091 with bad export cookie 3894440162831721284 [ 335.634917] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 335.638046] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 340.451268] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.451455] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 340.452351] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 340.452359] Lustre: Skipped 2 previous similar messages [ 340.468175] Lustre: Skipped 1 previous similar message [ 340.491255] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 342.056335] Lustre: server umount lustre-MDT0001 complete [ 346.065528] Lustre: server umount lustre-OST0000 complete [ 350.099895] Lustre: server umount lustre-OST0001 complete [ 356.672829] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [ 367.553420] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 403.345706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 403.570365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 403.610923] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 403.684510] Lustre: lustre-MDT0000: new disk, initializing [ 403.743827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 403.758622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 409.674830] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 422.816840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 422.969514] Lustre: 15081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 423.015937] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 423.019337] Lustre: Skipped 1 previous similar message [ 423.096025] Lustre: lustre-MDT0001: new disk, initializing [ 423.211400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 423.246339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 423.254932] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 427.624183] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 436.981829] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 443.708511] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 450.908186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 451.175745] Lustre: lustre-OST0000: new disk, initializing [ 451.179498] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 451.185453] Lustre: 17051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 451.285648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 452.425548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 452.438557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 452.532827] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 457.486358] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 467.072711] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 471.355818] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 471.553519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 475.534408] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 475.706734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 478.185601] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 478.194482] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 478.205184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 478.217257] Lustre: Skipped 1 previous similar message [ 478.229552] Lustre: Skipped 2 previous similar messages [ 483.305395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 483.320105] Lustre: Skipped 1 previous similar message [ 484.196286] Lustre: server umount lustre-OST0000 complete [ 488.423080] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 488.440878] Lustre: Skipped 1 previous similar message [ 493.537449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 493.543967] Lustre: Skipped 3 previous similar messages [ 493.775691] Lustre: server umount lustre-MDT0000 complete [ 497.187836] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787884253 with bad export cookie 3894440162831722817 [ 497.192190] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 497.203117] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 497.507469] Lustre: server umount lustre-MDT0001 complete [ 503.325789] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 22:30:57 (1787884257) [ 520.695422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 521.421180] LustreError: 19925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.495452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 525.506824] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 526.828482] LustreError: 19926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.941234] LustreError: 19925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 533.701169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 538.932106] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 547.238847] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 553.518858] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 556.111648] Lustre: Failing over lustre-MDT0000 [ 556.129040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.143241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 556.147415] Lustre: Skipped 1 previous similar message [ 556.164685] Lustre: Skipped 1 previous similar message [ 556.388523] Lustre: server umount lustre-MDT0000 complete [ 558.178359] LustreError: 19920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.197305] LustreError: 19920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 568.802558] LustreError: 19920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.826818] LustreError: 19920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 574.945101] Lustre: 3650:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787884314/real 1787884314] req@ffff8893c2580380 x1874732086622336/t0(0) o400->MGC192.168.203.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787884330 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 574.980980] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 577.742963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 585.193556] LustreError: 20653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 585.215984] LustreError: 20653:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 590.534271] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 590.820891] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 590.834126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 590.886886] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 600.799877] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 606.177871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 606.191140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 606.209360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 611.292613] Lustre: server umount lustre-MDT0000 complete [ 615.333461] LustreError: 22586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787884371 with bad export cookie 3894440162831724231 [ 615.335975] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 615.343702] LustreError: 22586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 615.806932] Lustre: server umount lustre-MDT0001 complete [ 625.616387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 626.110482] LustreError: 23262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 626.141318] LustreError: 23262:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 626.184848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 626.193094] Lustre: Skipped 2 previous similar messages [ 630.783729] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 640.098065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 645.151705] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 653.430535] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 660.038810] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 662.428981] Lustre: Failing over lustre-MDT0000 [ 662.796568] Lustre: server umount lustre-MDT0000 complete [ 665.506610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 665.534456] Lustre: Skipped 2 previous similar messages [ 681.962922] Lustre: 3647:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787884421/real 1787884421] req@ffff8894ff62e300 x1874732086651776/t0(0) o400->MGC192.168.203.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787884437 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 682.009894] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 683.300270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 691.255053] LustreError: 23257:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 691.304402] LustreError: 23257:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 691.663039] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 696.481427] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 696.807312] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 696.813911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 696.824778] Lustre: Skipped 1 previous similar message [ 696.846626] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 708.414435] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 715.235757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 715.240962] Lustre: Skipped 4 previous similar messages [ 719.437665] Lustre: server umount lustre-MDT0000 complete [ 723.793700] LustreError: 23244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787884479 with bad export cookie 3894440162831725169 [ 723.802507] LustreError: 23244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 723.803173] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 724.294111] Lustre: server umount lustre-MDT0001 complete [ 734.619776] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 22:34:48 (1787884488) [ 754.711408] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [ 765.756563] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 811.779470] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 825.703922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 826.083436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 826.144921] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 826.251652] Lustre: lustre-MDT0000: new disk, initializing [ 826.371440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 826.376842] Lustre: Skipped 2 previous similar messages [ 826.392604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 831.785485] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 844.889442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 844.995562] Lustre: 29660:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 845.203765] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 850.187916] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 862.111591] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 868.479329] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 876.998951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 877.385983] Lustre: 31629:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 878.817068] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 878.909843] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 878.943769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 878.964349] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 884.616360] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 894.331913] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 901.194681] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 930.277487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 930.286489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 930.301257] LustreError: Skipped 1 previous similar message [ 930.303741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 930.359089] Lustre: Skipped 3 previous similar messages [ 931.622335] Lustre: server umount lustre-OST0000 complete [ 937.441477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 950.240505] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 950.621965] Lustre: server umount lustre-MDT0000 complete [ 950.764600] LustreError: 29672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 950.810360] LustreError: 29672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 954.980320] LustreError: 29651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787884710 with bad export cookie 3894440162831725575 [ 954.982367] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 954.991122] LustreError: 29651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 955.507231] Lustre: server umount lustre-MDT0001 complete [ 969.571144] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 972.511797] Key type lgssc unregistered [ 972.943749] LNet: 33612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 972.961106] LNetError: 33612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 973.006367] LNet: Removed LNI 192.168.203.141@tcp [ 974.139260] Key type .llcrypt unregistered [ 974.141405] Key type ._llcrypt unregistered [ 1001.622867] Key type ._llcrypt registered [ 1001.624889] Key type .llcrypt registered [ 1001.713091] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 1003.009182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1003.059039] alg: No test for adler32 (adler32-zlib) [ 1004.082598] Lustre: Lustre: Build Version: 2.17.57_84_gd387c42 [ 1004.337134] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 1006.032188] Key type lgssc registered [ 1007.580867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1016.407680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1016.424395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1018.009510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1023.195577] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1027.561692] LustreError: 35025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1027.600024] LustreError: 35025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1032.408224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1032.675086] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1032.901439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1037.605775] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1045.263335] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1051.288452] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1058.342356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1058.758833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1062.969168] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1065.121595] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1072.992605] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1077.243927] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1089.784428] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 22:40:43 (1787884843) [ 1091.689216] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1093.603969] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1101.490234] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 22:40:54 (1787884854) [ 1110.912946] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 22:41:04 (1787884864) [ 1112.128446] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1123.318622] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1123.333569] Lustre: Skipped 2 previous similar messages [ 1125.365738] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 22:41:18 (1787884878) [ 1127.354781] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1127.358166] Lustre: Skipped 2 previous similar messages [ 1135.705507] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1135.715038] Lustre: Skipped 4 previous similar messages [ 1151.075800] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 22:41:44 (1787884904) [ 1154.189258] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1158.117472] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1168.967913] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 22:42:02 (1787884922) [ 1171.573158] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1171.576240] Lustre: Skipped 6 previous similar messages [ 1186.484106] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 22:42:19 (1787884939) [ 1205.228251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1205.229563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1205.253396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1205.283960] Lustre: Skipped 1 previous similar message [ 1210.343062] Lustre: server umount lustre-OST0000 complete [ 1215.464583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.495261] Lustre: Skipped 1 previous similar message [ 1215.500947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1215.511975] Lustre: Skipped 1 previous similar message [ 1220.429455] Lustre: server umount lustre-MDT0000 complete [ 1220.579155] LustreError: 35025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1220.610050] LustreError: 35025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1224.833101] LustreError: 35006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787884980 with bad export cookie 1174776409077797992 [ 1224.854880] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1224.858555] LustreError: 35006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1225.392898] Lustre: server umount lustre-MDT0001 complete [ 1237.450681] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 1240.230324] Key type lgssc unregistered [ 1240.559933] LNet: 42889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1240.565419] LNetError: 42889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1240.583849] LNet: Removed LNI 192.168.203.141@tcp [ 1241.358716] Key type .llcrypt unregistered [ 1241.361303] Key type ._llcrypt unregistered [ 1257.609298] Key type ._llcrypt registered [ 1257.612203] Key type .llcrypt registered [ 1257.705423] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [ 1272.520036] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 1273.598361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1273.695965] alg: No test for adler32 (adler32-zlib) [ 1274.796588] Lustre: Lustre: Build Version: 2.17.57_84_gd387c42 [ 1275.080837] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 1276.762817] Key type lgssc registered [ 1277.821747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1317.668505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1317.697594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1319.006281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1319.035161] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1319.115272] Lustre: lustre-MDT0000: new disk, initializing [ 1319.199991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1319.216810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1323.247817] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1334.309244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1334.436622] Lustre: 46040:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1334.482955] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1334.492530] Lustre: Skipped 1 previous similar message [ 1334.571696] Lustre: lustre-MDT0001: new disk, initializing [ 1334.662763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1334.724463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1334.748938] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1338.994287] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1348.577203] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1354.931291] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1361.744112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1362.063248] Lustre: lustre-OST0000: new disk, initializing [ 1362.066565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1362.079402] Lustre: 48010:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1362.206275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1367.941953] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1369.141896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1369.153498] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1369.241682] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1376.984311] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1381.301180] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1381.518718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1385.385572] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1385.550419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1389.539020] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1389.545168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1389.556961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1390.049459] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1390.060724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1390.070353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1390.080744] Lustre: Skipped 1 previous similar message [ 1393.413505] Lustre: server umount lustre-OST0000 complete [ 1400.291210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1400.303468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1400.312477] Lustre: Skipped 1 previous similar message [ 1400.332029] Lustre: Skipped 1 previous similar message [ 1403.306523] Lustre: server umount lustre-MDT0000 complete [ 1406.656857] LustreError: 49350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787885162 with bad export cookie 11947527084583139110 [ 1406.670468] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1406.671920] LustreError: 49350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1407.024716] Lustre: server umount lustre-MDT0001 complete [ 1416.504184] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 22:46:10 (1787885170) [ 1422.275309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1422.693887] LustreError: 50468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1422.711268] LustreError: 50468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1422.767518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1426.769663] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1427.939648] LustreError: 50469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1433.058369] LustreError: 50468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1434.214255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1434.464154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1438.367695] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1446.535740] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1452.894402] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1460.291948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1460.590836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1466.421502] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1473.560200] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1487.465287] LustreError: 53018:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1488.209101] Lustre: Modifying parameter general.timeout=1 in log params [ 1489.742779] Lustre: Modifying parameter general.timeout=3 in log params [ 1489.747875] Lustre: Skipped 1 previous similar message [ 1491.934469] Lustre: Modifying parameter general.timeout=6 in log params [ 1491.940355] Lustre: Skipped 2 previous similar messages [ 1496.136700] Lustre: Modifying parameter general.timeout=11 in log params [ 1496.144702] Lustre: Skipped 4 previous similar messages [ 1504.472051] Lustre: Modifying parameter general.timeout=21 in log params [ 1504.479187] Lustre: Skipped 9 previous similar messages [ 1521.228526] Lustre: Modifying parameter general.timeout=41 in log params [ 1521.232656] Lustre: Skipped 19 previous similar messages [ 1553.320562] Lustre: Modifying parameter general.timeout=75 in log params [ 1553.328655] Lustre: Skipped 33 previous similar messages [ 1616.425511] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 22:49:30 (1787885370) [ 1618.030324] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 in log params [ 1618.036506] Lustre: Skipped 56 previous similar messages [ 1618.880841] Lustre: 59963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1620.536412] Lustre: 60061:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1623.664590] Lustre: 60259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1623.670194] Lustre: 60259:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1653.140580] Lustre: 61835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1653.165095] Lustre: 61835:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1653.191518] LustreError: 61835:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1656.972707] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 22:50:10 (1787885410) [ 1670.113772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1670.129615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1670.141411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1673.835505] Lustre: server umount lustre-MDT0000 complete [ 1677.267053] LustreError: 50451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787885433 with bad export cookie 11947527084583139992 [ 1677.273803] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1677.279530] LustreError: 50451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1677.301986] LustreError: 50463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1677.308638] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1677.335470] Lustre: Skipped 2 previous similar messages [ 1677.351827] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1677.361283] Lustre: Skipped 3 previous similar messages [ 1677.648271] Lustre: server umount lustre-MDT0001 complete [ 1683.150735] Lustre: server umount lustre-OST0000 complete [ 1709.083482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1709.387497] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1709.745601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1714.800170] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1722.623063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1722.709925] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1722.749817] Lustre: 64364:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1722.960732] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1726.987846] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1733.969972] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1739.387676] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1747.000476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1747.191084] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1747.489534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1747.493204] Lustre: Skipped 1 previous similar message [ 1753.209362] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1761.056809] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1762.847870] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1762.857446] Lustre: Skipped 21 previous similar messages [ 1762.919960] Lustre: 66933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1769.954598] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1769.977322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1769.996284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1776.083163] Lustre: server umount lustre-OST0000 complete [ 1779.681132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1785.971706] Lustre: server umount lustre-MDT0000 complete [ 1789.378212] LustreError: 64358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787885545 with bad export cookie 11947527084583143002 [ 1789.381757] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1789.387141] LustreError: 64358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1789.689850] Lustre: server umount lustre-MDT0001 complete [ 1799.933512] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 1802.634209] Key type lgssc unregistered [ 1802.987776] LNet: 68139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.004426] LNetError: 68139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1803.041507] LNet: Removed LNI 192.168.203.141@tcp [ 1804.094844] Key type .llcrypt unregistered [ 1804.101209] Key type ._llcrypt unregistered [ 1830.407987] Key type ._llcrypt registered [ 1830.413904] Key type .llcrypt registered [ 1830.519663] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 1831.391052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1831.531264] alg: No test for adler32 (adler32-zlib) [ 1832.632866] Lustre: Lustre: Build Version: 2.17.57_84_gd387c42 [ 1832.974238] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 1834.640176] Key type lgssc registered [ 1835.910970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1844.019500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1844.032465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1845.512492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1849.409334] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1855.982655] LustreError: 69550:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1856.000701] LustreError: 69550:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1857.722715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1858.357821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1863.165430] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1870.905811] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1877.058680] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1883.386933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1884.077356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1890.300985] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1897.978951] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1909.754986] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1919.458950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1919.469254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1919.499424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1923.044755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1923.048358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1923.058706] Lustre: Skipped 1 previous similar message [ 1923.093345] Lustre: Skipped 1 previous similar message [ 1925.280681] Lustre: server umount lustre-MDT0000 complete [ 1928.165729] LustreError: 69544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1928.186899] LustreError: 69544:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1929.846350] LustreError: 69531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787885685 with bad export cookie 809277944757080117 [ 1929.862426] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1929.863276] LustreError: 69531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1930.241327] Lustre: server umount lustre-MDT0001 complete [ 1945.312313] Lustre: server umount lustre-OST0000 complete [ 1973.174203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1973.573383] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1973.622730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1973.910274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1978.261393] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1986.084175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1986.263501] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1986.308329] Lustre: 74424:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1986.851177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1992.416351] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2000.584328] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2006.595892] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2012.678184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2012.832622] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2013.241331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2019.428555] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2026.651781] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2028.147319] Lustre: 76999:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2030.097557] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2035.683096] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2035.690771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2035.692031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2035.692036] Lustre: Skipped 1 previous similar message [ 2035.717088] Lustre: Skipped 1 previous similar message [ 2040.804607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2040.820988] Lustre: Skipped 2 previous similar messages [ 2041.189938] Lustre: server umount lustre-OST0000 complete [ 2045.928191] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2045.942331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2045.942614] Lustre: Skipped 1 previous similar message [ 2045.957992] Lustre: Skipped 1 previous similar message [ 2051.046536] LustreError: 74435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2051.059731] LustreError: 74435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2051.109353] Lustre: server umount lustre-MDT0000 complete [ 2055.026149] LustreError: 74414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787885810 with bad export cookie 809277944757080957 [ 2055.027971] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2055.041139] LustreError: 74414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2055.423419] Lustre: server umount lustre-MDT0001 complete [ 2067.226470] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 2069.952610] Key type lgssc unregistered [ 2070.226817] LNet: 78207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2070.234216] LNetError: 78207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2070.260188] LNet: Removed LNI 192.168.203.141@tcp [ 2070.964086] Key type .llcrypt unregistered [ 2070.967347] Key type ._llcrypt unregistered [ 2095.317545] Key type ._llcrypt registered [ 2095.325379] Key type .llcrypt registered [ 2095.449475] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 2096.301235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2096.467245] alg: No test for adler32 (adler32-zlib) [ 2097.645246] Lustre: Lustre: Build Version: 2.17.57_84_gd387c42 [ 2097.910808] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 2099.560180] Key type lgssc registered [ 2100.632503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2108.474164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2108.508396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2109.999785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2114.019860] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2120.169768] LustreError: 79615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2120.192070] LustreError: 79615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2123.248857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2124.004292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2129.272406] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2138.741170] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2145.414247] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2153.143033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2153.848142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2160.200802] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2167.325574] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2176.030725] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2176.888490] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2178.645338] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2178.656265] Lustre: Skipped 1 previous similar message [ 2181.143696] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2181.148730] Lustre: Skipped 2 previous similar messages [ 2185.989760] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2185.996546] Lustre: Skipped 5 previous similar messages [ 2194.161454] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2194.167867] Lustre: Skipped 9 previous similar messages [ 2210.767785] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2210.772891] Lustre: Skipped 19 previous similar messages [ 2243.459646] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2243.464259] Lustre: Skipped 35 previous similar messages [ 2275.084279] Lustre: 87715:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2277.810143] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 23:00:31 (1787886031) [ 2279.529367] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2281.501183] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 23:00:35 (1787886035) [ 2287.080318] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2287.081051] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2287.095183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2287.104076] Lustre: Skipped 1 previous similar message [ 2287.123088] Lustre: Skipped 1 previous similar message [ 2292.197205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2292.204921] Lustre: Skipped 1 previous similar message [ 2292.600719] Lustre: server umount lustre-OST0000 complete [ 2297.319446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2297.341615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2297.346209] Lustre: Skipped 1 previous similar message [ 2302.433308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2302.445267] Lustre: Skipped 1 previous similar message [ 2302.936431] Lustre: server umount lustre-MDT0000 complete [ 2307.450710] LustreError: 79594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787886063 with bad export cookie 3561020326955204766 [ 2307.453387] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2307.469680] LustreError: 79594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2307.820617] Lustre: server umount lustre-MDT0001 complete [ 2320.158986] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 2323.069630] Key type lgssc unregistered [ 2323.390759] LNet: 88976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2323.397870] LNetError: 88976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2323.423157] LNet: Removed LNI 192.168.203.141@tcp [ 2324.217528] Key type .llcrypt unregistered [ 2324.219747] Key type ._llcrypt unregistered [ 2329.030669] Key type ._llcrypt registered [ 2329.034365] Key type .llcrypt registered [ 2329.173622] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2334.538455] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules [ 2335.975825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2336.125011] alg: No test for adler32 (adler32-zlib) [ 2337.467293] Lustre: Lustre: Build Version: 2.17.57_84_gd387c42 [ 2337.809472] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 2339.499317] Key type lgssc registered [ 2340.609619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2342.211153] LustreError: 90224:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2363.592921] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 2365.875150] LustreError: 90224:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2373.914924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2373.930497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2375.366536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2379.935445] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2385.892399] LustreError: 91252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2385.913054] LustreError: 91252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2389.009564] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 23:02:22 (1787886142) [ 2391.009287] LustreError: 91251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2394.014966] LustreError: 91283:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2394.035234] LustreError: 91283:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2394.042802] LustreError: 91283:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2394.329470] Lustre: server umount lustre-MDT0000 complete [ 2408.668746] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 2411.360754] Key type lgssc unregistered [ 2411.638800] LNet: 92774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2411.644531] LNetError: 92774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2411.664889] LNet: Removed LNI 192.168.203.141@tcp [ 2412.673120] Key type .llcrypt unregistered [ 2412.675825] Key type ._llcrypt unregistered [ 2430.316269] Key type ._llcrypt registered [ 2430.321236] Key type .llcrypt registered [ 2430.429987] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 2431.189521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2431.229792] alg: No test for adler32 (adler32-zlib) [ 2432.247689] Lustre: Lustre: Build Version: 2.17.57_84_gd387c42 [ 2432.476851] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 2434.152160] Key type lgssc registered [ 2435.098877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2442.841126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2442.878688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2444.346747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2449.094731] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2454.502339] LustreError: 93987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2454.521637] LustreError: 93987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2457.684336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2458.296293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2462.486669] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2469.529665] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2474.728525] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2481.128488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2481.887582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2487.997426] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2496.120960] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2547.208216] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 23:05:00 (1787886300) [ 2549.336905] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2551.884954] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 23:05:05 (1787886305) [ 2555.364079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2555.374769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2555.400508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2556.897568] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2556.901867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2556.927397] Lustre: Skipped 1 previous similar message [ 2556.936118] Lustre: Skipped 2 previous similar messages [ 2561.523874] Lustre: server umount lustre-MDT0000 complete [ 2562.020201] LustreError: 93986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2562.070025] LustreError: 93986:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2565.497587] LustreError: 93969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787886321 with bad export cookie 3612498779541049674 [ 2565.504944] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2565.515801] LustreError: 93969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2566.032338] Lustre: server umount lustre-MDT0001 complete [ 2581.325169] Lustre: server umount lustre-OST0000 complete [ 2595.076859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2595.653664] LustreError: 98285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2595.896136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2600.734356] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2600.929292] LustreError: 98286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2606.053460] LustreError: 98285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2610.490449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2611.376027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2616.689295] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2624.153504] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2629.614390] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2643.546622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2643.767085] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2643.779724] LustreError: 98273:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2643.834894] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2643.844113] LustreError: 100372:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2643.858705] LustreError: 100372:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2643.866073] LustreError: 100372:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2643.978716] Lustre: server umount lustre-OST0000 complete [ 2643.985274] LustreError: 100372:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2652.817949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2653.107716] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2653.118752] LustreError: 98273:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2653.148130] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2653.156919] LustreError: 100837:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2653.163684] LustreError: 100837:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2653.172090] LustreError: 100837:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2653.270449] Lustre: server umount lustre-OST0000 complete [ 2653.272596] LustreError: 100837:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2661.812364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2662.008196] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2662.013252] Lustre: Found index 0 for lustre-OST0000, updating log [ 2662.023478] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2662.124601] Lustre: lustre-OST0000: new disk, initializing [ 2662.139423] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2662.154391] Lustre: 101370:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2662.408457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2668.128848] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2671.375880] Lustre: Failing over lustre-OST0000 [ 2671.529652] Lustre: server umount lustre-OST0000 complete [ 2675.683035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2675.686128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2675.707897] Lustre: Skipped 1 previous similar message [ 2680.807091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2680.819656] Lustre: Skipped 2 previous similar messages [ 2685.011916] Lustre: server umount lustre-MDT0000 complete [ 2686.947365] LustreError: 98281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.983364] LustreError: 98281:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2689.131391] LustreError: 100391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787886444 with bad export cookie 3612498779541052411 [ 2689.153183] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2689.156907] LustreError: 100391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2689.613645] Lustre: server umount lustre-MDT0001 complete [ 2698.625302] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 23:07:32 (1787886452) [ 2713.504810] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 2723.088297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2723.466718] LustreError: 103714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2723.561648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2728.257107] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2736.424331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2741.110298] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2743.934342] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2750.919147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2751.525513] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2758.759541] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2761.765742] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2761.776398] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2761.807651] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2766.822191] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2766.840644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2768.443414] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2768.539982] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2768.630508] Lustre: lustre-OST0001: new disk, initializing [ 2768.642148] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2768.652648] Lustre: 106033:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2768.868255] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:02 [ 2768.902533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2768.904212] Lustre: lustre-OST0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted. [ 2768.919864] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2768.937453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2769.025499] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2769.036009] Lustre: Skipped 2 previous similar messages [ 2776.215390] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2778.791358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2778.825612] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2779.062523] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2784.602867] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2793.380276] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2798.573419] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2798.584525] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2798.601588] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2798.610464] Lustre: Skipped 2 previous similar messages [ 2804.714929] LustreError: 105072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2804.744171] LustreError: 105072:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2804.837983] Lustre: server umount lustre-MDT0001 complete [ 2815.335648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2815.421702] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2815.430551] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2815.482819] Lustre: 103702:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2815.703106] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.41@tcp (not set up) [ 2815.718576] Lustre: Skipped 4 previous similar messages [ 2816.955199] LustreError: 107572:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2816.973711] LustreError: 107572:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2816.989839] LustreError: 103707:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2817.188708] LustreError: 107582:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.141@tcp: cfg command failed: rc = -17 [ 2817.195233] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2817.195233] [ 2821.609904] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2821.781825] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2830.408657] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 23:09:44 (1787886584) [ 2846.857314] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 2864.530245] Lustre: 109650:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2885.427702] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2888.530832] Lustre: 110784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2969.569108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2969.590686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2969.607845] Lustre: Skipped 2 previous similar messages [ 2969.615447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2974.448681] Lustre: server umount lustre-MDT0000 complete [ 2975.201140] LustreError: 103711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2975.239493] LustreError: 103711:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 2978.577299] LustreError: 103695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787886734 with bad export cookie 3612498779541053419 [ 2978.585220] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2978.592374] LustreError: 103695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2979.173542] Lustre: server umount lustre-MDT0001 complete [ 2993.558098] Lustre: server umount lustre-OST0000 complete [ 2996.512098] Lustre: 93298:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787886736/real 1787886736] req@ffff8893c857f480 x1874734547821824/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787886752 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2996.551594] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2996.572088] Lustre: Skipped 3 previous similar messages [ 2997.437649] Lustre: server umount lustre-OST0001 complete [ 3003.843414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3015.958270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3025.176571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3034.296251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3052.742716] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 3063.182438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3063.205784] Lustre: lustre-MDT0000: reset Object Index mappings [ 3063.620374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3063.626670] Lustre: Skipped 1 previous similar message [ 3068.425232] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3075.791076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3075.820077] Lustre: lustre-MDT0001: reset Object Index mappings [ 3080.541555] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3083.211380] Lustre: 115881:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3089.102246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3095.519481] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3101.061114] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 3103.861864] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3108.821824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3108.828718] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3108.973851] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3111.050904] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3118.035849] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3121.524190] Lustre: 117757:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3169.078890] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 23:15:22 (1787886922) [ 3195.876993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3195.895757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3195.896456] Lustre: Skipped 2 previous similar messages [ 3195.907735] Lustre: Skipped 7 previous similar messages [ 3201.531673] Lustre: server umount lustre-MDT0000 complete [ 3205.520858] LustreError: 114714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787886961 with bad export cookie 3612498779541100515 [ 3205.533564] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3205.535442] LustreError: 114714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3206.117163] LustreError: 117786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3206.134175] LustreError: 117786:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3212.079596] Lustre: server umount lustre-MDT0001 complete [ 3222.212693] Lustre: server umount lustre-OST0000 complete [ 3233.472124] Lustre: server umount lustre-OST0001 complete [ 3240.695497] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [ 3250.575661] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 3285.651513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3285.932383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3285.962990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3286.071511] Lustre: lustre-MDT0000: new disk, initializing [ 3286.149179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3286.154335] Lustre: Skipped 3 previous similar messages [ 3286.197250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3291.106359] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3297.970543] Lustre: server umount lustre-MDT0000 complete [ 3307.461755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3307.721534] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3307.747234] Lustre: 122025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3307.763145] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3312.341441] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3317.163673] Lustre: server umount lustre-MDT0000 complete [ 3325.814919] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 23:17:59 (1787887079) [ 3327.349409] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3329.075432] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 23:18:02 (1787887082) [ 3344.888330] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 3353.761374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3358.957198] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3367.705085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3367.876485] Lustre: 124072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3367.951170] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3367.956816] Lustre: Skipped 1 previous similar message [ 3368.049415] Lustre: lustre-MDT0001: new disk, initializing [ 3368.259781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3368.280072] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3373.576625] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3377.820220] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3383.862510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3384.172704] Lustre: lustre-OST0000: new disk, initializing [ 3384.176870] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3384.185833] Lustre: 125651:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3385.734363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3385.741285] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3385.929783] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3391.139959] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3402.115716] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3402.428701] Lustre: lustre-OST0001: new disk, initializing [ 3402.441707] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3402.458362] Lustre: 126522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3404.425045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3404.457214] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3404.568706] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3410.237859] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3421.320722] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3425.552277] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3798.890614] Lustre: 132934:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3798.900930] Lustre: 132934:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3798.909958] Lustre: 132934:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3798.921431] Lustre: 132934:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3798.930168] Lustre: 132934:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3798.936466] Lustre: 132934:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4040.651665] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:29:54 (1787887794) [ 4054.504201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4054.515856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4054.533370] Lustre: Skipped 3 previous similar messages [ 4054.546085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4054.556109] Lustre: Skipped 8 previous similar messages [ 4056.676283] Lustre: server umount lustre-MDT0000 complete [ 4059.958381] LustreError: 132962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787887815 with bad export cookie 3612498779541167995 [ 4059.958884] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4059.976011] LustreError: 132962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4060.129629] LustreError: 127327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4060.132196] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4060.149579] LustreError: 127327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4060.164657] Lustre: Skipped 4 previous similar messages [ 4064.167300] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4064.182762] Lustre: Skipped 7 previous similar messages [ 4066.571596] Lustre: server umount lustre-MDT0001 complete [ 4076.333687] Lustre: server umount lustre-OST0000 complete [ 4085.968204] Lustre: server umount lustre-OST0001 complete [ 4092.723684] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [ 4103.369170] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 4138.634116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4138.881643] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4138.921199] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4139.026440] Lustre: lustre-MDT0000: new disk, initializing [ 4139.123258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4139.127984] Lustre: Skipped 5 previous similar messages [ 4139.138649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4143.778401] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 4153.860595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4153.985951] Lustre: 135976:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4154.058240] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4154.062083] Lustre: Skipped 1 previous similar message [ 4154.133933] Lustre: lustre-MDT0001: new disk, initializing [ 4154.279141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4154.296669] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4157.744762] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 4166.215863] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4171.367494] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4177.242262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4177.492740] Lustre: lustre-OST0000: new disk, initializing [ 4177.498602] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4177.505446] Lustre: 137955:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4178.625721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4178.641415] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4178.750430] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4183.108523] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 4191.676790] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4199.388647] Lustre: lustre-MDD0000: changelog on [ 4201.809253] Lustre: lustre-MDD0001: changelog on [ 5935.624468] Lustre: lustre-MDD0001: changelog off [ 5937.790725] Lustre: 137570:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5938.631900] Lustre: lustre-MDD0000: changelog off [ 5950.438841] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5950.464472] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5950.466780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5950.476287] Lustre: Skipped 1 previous similar message [ 5955.554827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5955.560896] Lustre: Skipped 1 previous similar message [ 5960.164220] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5960.380556] Lustre: server umount lustre-OST0000 complete [ 5965.794662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5965.810505] Lustre: Skipped 1 previous similar message [ 5965.820569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5965.824696] Lustre: Skipped 1 previous similar message [ 5970.775617] Lustre: server umount lustre-MDT0000 complete [ 5970.921419] LustreError: 136882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5970.949986] LustreError: 136882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5975.246187] LustreError: 135969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787889731 with bad export cookie 3612498779541942930 [ 5975.252962] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5975.266123] LustreError: 135969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5975.683865] Lustre: server umount lustre-MDT0001 complete [ 5989.110537] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 5992.636406] Key type lgssc unregistered [ 5993.122981] LNet: 147071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5993.133300] LNetError: 147071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5993.160917] LNet: Removed LNI 192.168.203.141@tcp [ 5994.148179] Key type .llcrypt unregistered [ 5994.150064] Key type ._llcrypt unregistered [ 6014.845572] Key type ._llcrypt registered [ 6014.851572] Key type .llcrypt registered [ 6014.988591] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [ 6030.841897] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 6032.266375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6032.312959] alg: No test for adler32 (adler32-zlib) [ 6033.352964] Lustre: Lustre: Build Version: 2.17.57_84_gd387c42 [ 6033.561216] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 6035.256281] Key type lgssc registered [ 6037.311296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6075.595913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6075.635376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6076.896881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6076.941116] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6077.058237] Lustre: lustre-MDT0000: new disk, initializing [ 6077.165647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6077.195145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6082.240695] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6092.204481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6092.319622] Lustre: 150272:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6092.396224] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6092.403020] Lustre: Skipped 1 previous similar message [ 6092.481851] Lustre: lustre-MDT0001: new disk, initializing [ 6092.607143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6092.638869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6092.652887] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6096.988408] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6106.504854] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6112.190986] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6118.653497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6118.841762] Lustre: lustre-OST0000: new disk, initializing [ 6118.844951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6118.850186] Lustre: 152237:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6118.936901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6125.058583] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6127.147194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6127.154624] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6127.259192] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6133.469797] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6140.153696] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_module obdecho/obdecho [ 6141.882409] LustreError: 153241:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6141.892583] LustreError: 153241:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6143.520051] LustreError: 153337:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6154.210157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6154.215466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6154.226318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6158.308140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6158.309582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6158.325948] Lustre: Skipped 1 previous similar message [ 6158.344632] Lustre: Skipped 2 previous similar messages [ 6160.378945] Lustre: server umount lustre-MDT0000 complete [ 6163.374258] LustreError: 150262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787889919 with bad export cookie 5230050570248511282 [ 6163.375892] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6163.389878] LustreError: 150262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6163.430231] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6163.440567] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6163.668502] Lustre: server umount lustre-MDT0001 complete [ 6169.728445] Lustre: server umount lustre-OST0000 complete [ 6190.495940] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [ 6203.071608] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 6238.085103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6238.296425] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6238.318806] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6238.403530] Lustre: lustre-MDT0000: new disk, initializing [ 6238.474576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6238.506819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6242.497495] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6253.618575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6253.761512] Lustre: 157051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6253.842324] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6253.849070] Lustre: Skipped 1 previous similar message [ 6253.944269] Lustre: lustre-MDT0001: new disk, initializing [ 6254.050573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6254.100243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6254.112621] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6258.061431] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6267.283099] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6272.525853] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6279.521843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6279.806774] Lustre: lustre-OST0000: new disk, initializing [ 6279.811980] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6279.823712] Lustre: 159018:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6279.937457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6286.691369] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6289.499072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6289.519535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6289.609501] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6296.165230] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6301.308682] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6301.523947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6306.052803] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6306.233326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6310.376532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6310.386952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6310.402355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6310.407089] Lustre: Skipped 1 previous similar message [ 6314.311629] Lustre: server umount lustre-OST0000 complete [ 6320.614613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6320.615663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6320.633622] LustreError: Skipped 1 previous similar message [ 6320.637937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6320.661444] Lustre: Skipped 1 previous similar message [ 6320.671121] Lustre: Skipped 1 previous similar message [ 6323.859314] Lustre: server umount lustre-MDT0000 complete [ 6325.734036] LustreError: 160357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6325.761526] LustreError: 160357:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6327.627266] LustreError: 159017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787890083 with bad export cookie 5230050570248512241 [ 6327.627852] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6327.649056] LustreError: 159017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6327.957767] Lustre: server umount lustre-MDT0001 complete [ 6331.316706] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 00:08:05 (1787890085) [ 6332.584864] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6334.312901] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 00:08:08 (1787890088) [ 6340.639953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6341.011405] LustreError: 161431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6341.076819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6344.602899] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6346.220113] LustreError: 161432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6350.312668] LustreError: 161431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6351.870228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6356.433861] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6363.142959] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6367.774601] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6373.415060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6373.746916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6373.756364] Lustre: Skipped 1 previous similar message [ 6379.117795] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6385.175685] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6396.129518] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.41@tcp (stopping) [ 6396.134684] Lustre: Skipped 1 previous similar message [ 6396.388752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6396.406781] Lustre: Skipped 2 previous similar messages [ 6398.091364] Lustre: server umount lustre-MDT0000 complete [ 6401.235812] LustreError: 162842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6401.294312] LustreError: 161412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787890157 with bad export cookie 5230050570248513123 [ 6401.295722] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6401.321186] LustreError: 161412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6407.648321] Lustre: server umount lustre-MDT0001 complete [ 6413.814364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6425.948203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6426.168447] LustreError: 165272:0:(llog.c:1655:llog_backup()) MGC192.168.203.141@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6426.174342] Lustre: 165272:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.141@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6431.203565] LustreError: 165277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6431.218305] LustreError: 165277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6448.332438] LustreError: 165277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6448.349443] LustreError: 165277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6467.048987] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6467.064156] Lustre: Skipped 6 previous similar messages [ 6481.892136] LustreError: 165279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6481.918952] LustreError: 165279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 22 previous similar messages [ 6487.521541] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6487.540206] Lustre: 165272:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.141@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6528.510982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6532.685260] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6533.612444] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6540.128901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6546.914552] LustreError: 165278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6546.934866] LustreError: 165278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 45 previous similar messages [ 6550.393405] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6552.554805] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6555.794366] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6557.009162] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6561.107370] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6562.298243] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6577.947563] Lustre: server umount lustre-MDT0000 complete [ 6578.147457] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6578.154383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6578.157593] Lustre: Skipped 2 previous similar messages [ 6580.794044] LustreError: 162166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787890336 with bad export cookie 5230050570248519577 [ 6580.797936] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6580.803360] LustreError: 162166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6581.021173] Lustre: server umount lustre-MDT0001 complete [ 6594.093252] Lustre: server umount lustre-OST0000 complete [ 6609.917527] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [ 6617.360553] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 6645.914204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6646.117887] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6646.144984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6646.199399] Lustre: lustre-MDT0000: new disk, initializing [ 6646.256371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6649.919832] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6658.485258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6658.556548] Lustre: 170375:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6658.797710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6658.800619] Lustre: Skipped 2 previous similar messages [ 6658.846700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6662.819255] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6671.250387] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6676.262098] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6682.938620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6683.129832] Lustre: lustre-OST0000: new disk, initializing [ 6683.133832] Lustre: Skipped 1 previous similar message [ 6683.136452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6683.148314] Lustre: Skipped 2 previous similar messages [ 6683.161762] Lustre: 172342:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6688.581292] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6693.465273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6693.471455] Lustre: Skipped 1 previous similar message [ 6693.476038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6693.576212] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6697.115568] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6701.354439] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6701.582592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6705.710513] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6705.895054] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6709.227219] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6709.228334] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6709.229820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6709.259364] Lustre: Skipped 2 previous similar messages [ 6713.720324] Lustre: server umount lustre-OST0000 complete [ 6723.448907] Lustre: server umount lustre-MDT0000 complete [ 6724.577507] LustreError: 170381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6724.589123] LustreError: 170381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6726.257681] LustreError: 171168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787890482 with bad export cookie 5230050570248536412 [ 6726.266195] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6726.269459] LustreError: 171168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6726.497701] Lustre: server umount lustre-MDT0001 complete [ 6734.838514] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 00:14:48 (1787890488) [ 6741.593420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6746.574928] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6755.096298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6759.769816] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6766.758787] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6772.033339] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6779.199607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6786.933229] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6795.153038] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6805.078317] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 00:15:58 (1787890558) [ 6811.624344] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6811.630341] LustreError: Skipped 1 previous similar message [ 6811.633624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6811.641091] Lustre: Skipped 2 previous similar messages [ 6811.644785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6811.647913] Lustre: Skipped 3 previous similar messages [ 6814.955070] Lustre: server umount lustre-OST0000 complete [ 6824.721348] Lustre: server umount lustre-MDT0000 complete [ 6827.962682] LustreError: 174734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787890583 with bad export cookie 5230050570248537294 [ 6827.967522] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6827.971147] LustreError: 174734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6828.323912] Lustre: server umount lustre-MDT0001 complete [ 6837.400448] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 6840.298506] Key type lgssc unregistered [ 6840.656175] LNet: 178476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6840.667641] LNetError: 178476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6840.696728] LNet: Removed LNI 192.168.203.141@tcp [ 6841.555209] Key type .llcrypt unregistered [ 6841.557431] Key type ._llcrypt unregistered [ 6859.743985] Key type ._llcrypt registered [ 6859.747854] Key type .llcrypt registered [ 6859.858801] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 6860.967976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6861.054264] alg: No test for adler32 (adler32-zlib) [ 6862.078519] Lustre: Lustre: Build Version: 2.17.57_84_gd387c42 [ 6862.290096] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 6863.962690] Key type lgssc registered [ 6865.038043] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6871.835081] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6871.851178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6887.392538] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6892.513466] LustreError: 179714:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.141@tcp: failed processing log, type 4: rc = -110 [ 6897.634102] LustreError: 179714:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6897.655116] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6897.664412] LustreError: MGC192.168.203.141@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6897.672849] LustreError: 179714:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6897.681353] LustreError: 179714:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 6897.687239] LustreError: 179714:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 6897.725388] Lustre: server umount lustre-OST0000 complete [ 6897.728526] LustreError: 179714:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6905.491246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6906.829872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6911.255855] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6917.090708] LustreError: 180213:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6917.106953] LustreError: 180213:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6919.035272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6919.376175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6922.716390] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6929.035519] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6933.719348] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6939.941035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6940.107175] LustreError: 182030:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6940.113691] Lustre: 182030:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.141@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6940.279769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6945.453132] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6952.696891] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6955.489284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6955.496067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6955.525532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6957.027903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6957.030393] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6957.046879] Lustre: Skipped 1 previous similar message [ 6957.070299] Lustre: Skipped 1 previous similar message [ 6961.693568] Lustre: server umount lustre-MDT0000 complete [ 6962.146486] LustreError: 180213:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6962.168840] LustreError: 180213:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6965.815415] LustreError: 180193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787890721 with bad export cookie 3331406332386376214 [ 6965.819947] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6965.836454] LustreError: 180193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6966.221519] Lustre: server umount lustre-MDT0001 complete [ 6980.684299] Lustre: server umount lustre-OST0000 complete [ 6989.513909] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [ 7000.445710] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 7037.513271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7037.781441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7037.822781] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7037.930371] Lustre: lustre-MDT0000: new disk, initializing [ 7038.025839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7038.060862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7042.564294] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 7052.338138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7052.478861] Lustre: 185555:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7052.527852] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7052.531910] Lustre: Skipped 1 previous similar message [ 7052.641477] Lustre: lustre-MDT0001: new disk, initializing [ 7052.741506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7052.795301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7052.807670] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7057.202236] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 7065.975745] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7070.841620] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7077.096563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7077.494912] Lustre: lustre-OST0000: new disk, initializing [ 7077.504497] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7077.521511] Lustre: 187520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7077.632445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7083.713572] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 7084.628497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7084.643317] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7084.773990] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7092.678964] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7096.825205] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7097.073473] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7101.504800] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7101.708879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7105.509766] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7105.514571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7105.516625] LustreError: Skipped 1 previous similar message [ 7105.538414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7105.538421] Lustre: Skipped 1 previous similar message [ 7109.892753] Lustre: server umount lustre-OST0000 complete [ 7114.209205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7114.225379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7114.241049] Lustre: Skipped 1 previous similar message [ 7114.255235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7114.267705] Lustre: Skipped 1 previous similar message [ 7119.613122] Lustre: server umount lustre-MDT0000 complete [ 7120.872618] LustreError: 186437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7120.884989] LustreError: 186437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7123.132178] LustreError: 185547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787890878 with bad export cookie 3331406332386376935 [ 7123.141929] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7123.146902] LustreError: 185547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7123.471922] Lustre: server umount lustre-MDT0001 complete [ 7131.503670] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 00:21:25 (1787890885) [ 7147.241599] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 7149.600150] Key type lgssc unregistered [ 7149.837741] LNet: 190464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7149.842931] LNetError: 190464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7149.859985] LNet: Removed LNI 192.168.203.141@tcp [ 7150.543228] Key type .llcrypt unregistered [ 7150.547139] Key type ._llcrypt unregistered [ 7165.548672] Key type ._llcrypt registered [ 7165.550595] Key type .llcrypt registered [ 7165.660477] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 7166.705365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7166.766232] alg: No test for adler32 (adler32-zlib) [ 7167.774438] Lustre: Lustre: Build Version: 2.17.57_84_gd387c42 [ 7168.001061] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 7169.634315] Key type lgssc registered [ 7170.858111] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7177.786423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7177.809046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7179.237485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7183.137910] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 7189.477027] LustreError: 191674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7189.501234] LustreError: 191674:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7191.046670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7191.496368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7195.600516] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 7202.065964] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7206.638433] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7213.048134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7213.188923] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7213.191281] LustreError: 193536:0:(llog.c:1696:llog_backup()) MGC192.168.203.141@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7213.206840] Lustre: 193536:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.141@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7213.323295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7218.125458] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 7224.288562] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7227.361933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7227.373496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7227.392863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7229.923950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7229.925287] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7229.937721] Lustre: Skipped 1 previous similar message [ 7229.948466] Lustre: Skipped 1 previous similar message [ 7232.233996] Lustre: server umount lustre-MDT0000 complete [ 7235.042424] LustreError: 191674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7235.059591] LustreError: 191674:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7235.396085] LustreError: 191653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787890991 with bad export cookie 7815754045765517635 [ 7235.400182] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7235.407205] LustreError: 191653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7235.719866] Lustre: server umount lustre-MDT0001 complete [ 7248.571389] Lustre: server umount lustre-OST0000 complete [ 7259.071211] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [ 7269.583342] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 7300.430981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7300.639979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7300.667768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7300.744068] Lustre: lustre-MDT0000: new disk, initializing [ 7300.813427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7300.845482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7304.153462] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 7314.265560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7314.402560] Lustre: 197058:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7314.441429] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7314.444296] Lustre: Skipped 1 previous similar message [ 7314.458268] LustreError: 197840:0:(llog.c:1696:llog_backup()) MGC192.168.203.141@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7314.476366] LustreError: 197840:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7314.492581] Lustre: 197840:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.141@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7314.547250] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7314.550373] Lustre: Skipped 1 previous similar message [ 7314.634393] Lustre: lustre-MDT0001: new disk, initializing [ 7314.732808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7314.786619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7314.807864] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7319.361615] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 7327.713623] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7332.676808] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7338.318390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7338.547478] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7338.550844] Lustre: Skipped 1 previous similar message [ 7338.553666] LustreError: 199003:0:(llog.c:1696:llog_backup()) MGC192.168.203.141@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7338.559152] LustreError: 199003:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7338.564136] Lustre: 199003:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.141@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7338.571478] Lustre: 199003:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7338.589952] Lustre: lustre-OST0000: new disk, initializing [ 7338.595312] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7338.600994] Lustre: 199028:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7338.673468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7344.448287] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 7345.194075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7345.204647] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7345.302593] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7353.673672] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7357.551284] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7357.742518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7362.040598] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7362.257357] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7366.113729] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7366.124527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7366.127320] LustreError: Skipped 1 previous similar message [ 7366.128850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7366.154663] Lustre: Skipped 1 previous similar message [ 7371.239712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7371.244559] Lustre: Skipped 2 previous similar messages [ 7376.358398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7376.369782] Lustre: Skipped 1 previous similar message [ 7378.400165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7378.594532] Lustre: server umount lustre-OST0000 complete [ 7386.595495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7386.600787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7386.611155] Lustre: Skipped 1 previous similar message [ 7386.621314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7388.539687] Lustre: server umount lustre-MDT0000 complete [ 7391.723689] LustreError: 200570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7391.752772] LustreError: 200570:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7392.424899] LustreError: 197050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787891148 with bad export cookie 7815754045765518363 [ 7392.438373] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7392.446927] LustreError: 197050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7393.138203] Lustre: server umount lustre-MDT0001 complete [ 7404.562870] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 00:25:57 (1787891157) [ 7420.085231] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 7429.660521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7430.065128] LustreError: 202157:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7430.115085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7433.813698] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 7435.238525] LustreError: 202158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7441.854132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7447.123462] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 7450.297919] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7458.765047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7465.367084] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 7473.168652] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7473.324176] Lustre: lustre-OST0001: new disk, initializing [ 7473.327415] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7473.332613] Lustre: 204461:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7473.412897] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7473.419760] Lustre: Skipped 2 previous similar messages [ 7477.443857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7477.451157] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7477.596369] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7479.605847] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 7488.986919] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7495.663957] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7497.367642] Lustre: DEBUG MARKER: ADD OST3 [ 7501.892913] loop: module loaded [ 7501.975164] loop0: detected capacity change from 0 to 204800000 [ 7502.184120] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7502.197802] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7502.215379] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7507.458210] loop0: detected capacity change from 0 to 204800000 [ 7508.899819] Lustre: DEBUG MARKER: STOP OST3 [ 7512.802785] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7512.946611] Lustre: lustre-OST0003: new disk, initializing [ 7512.950597] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7512.956456] Lustre: 206422:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7514.500358] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7514.690235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7514.699150] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7514.744279] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7516.516658] LustreError: 206583:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7517.348077] Lustre: Failing over lustre-OST0003 [ 7517.511996] Lustre: server umount lustre-OST0003 complete [ 7519.532697] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 7519.717982] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7519.722721] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7519.740354] LustreError: 204707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7519.749833] Lustre: Skipped 1 previous similar message [ 7519.787813] LustreError: 204707:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7529.147572] Lustre: DEBUG MARKER: START OST3 again [ 7535.475371] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7535.817760] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7535.832976] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7537.553425] Lustre: lustre-OST0003: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 7537.558271] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7541.740752] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 7561.823050] Lustre: Failing over lustre-OST0003 [ 7561.913517] Lustre: server umount lustre-OST0003 complete [ 7563.238599] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7563.250395] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7563.276240] LustreError: 203641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7563.304370] LustreError: 203641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 7580.642315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7580.657334] LustreError: Skipped 1 previous similar message [ 7580.664393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7580.678883] Lustre: Skipped 1 previous similar message [ 7580.684523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7580.691073] Lustre: Skipped 1 previous similar message [ 7585.715608] Lustre: server umount lustre-MDT0000 complete [ 7589.139724] LustreError: 202139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787891344 with bad export cookie 7815754045765519245 [ 7589.143352] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7589.155091] LustreError: 202139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7595.665877] Lustre: server umount lustre-MDT0001 complete [ 7599.523971] Lustre: server umount lustre-OST0000 complete [ 7603.246814] Lustre: server umount lustre-OST0001 complete [ 7609.803965] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [ 7617.098716] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 7654.500698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7654.746830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7654.775621] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7654.883403] Lustre: lustre-MDT0000: new disk, initializing [ 7654.977393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7654.982521] Lustre: Skipped 2 previous similar messages [ 7655.002240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7659.584690] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 7669.987963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7670.124976] Lustre: 211922:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7670.461880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7674.818939] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 7683.049925] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7688.876994] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7696.261265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7696.624618] Lustre: lustre-OST0000: new disk, initializing [ 7696.635307] Lustre: Skipped 1 previous similar message [ 7696.641678] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7696.646107] Lustre: Skipped 2 previous similar messages [ 7696.653432] Lustre: 213891:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7698.566693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7698.584988] Lustre: Skipped 1 previous similar message [ 7698.603515] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7698.752418] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7703.131228] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 7711.606969] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7715.766101] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7715.963543] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7720.171889] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7720.345425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7724.516774] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7724.519274] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7724.524081] LustreError: Skipped 1 previous similar message [ 7724.528470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7724.535438] Lustre: Skipped 6 previous similar messages [ 7724.550794] Lustre: Skipped 9 previous similar messages [ 7736.288136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7736.425717] Lustre: server umount lustre-OST0000 complete [ 7746.564053] Lustre: server umount lustre-MDT0000 complete [ 7749.956806] LustreError: 212717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787891505 with bad export cookie 7815754045765521541 [ 7749.973034] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7749.980234] LustreError: 212717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7750.555337] Lustre: server umount lustre-MDT0001 complete [ 7754.902951] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 00:31:48 (1787891508) [ 7771.843672] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [ 7782.460621] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 7818.267994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7818.555115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7818.620614] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7818.727102] Lustre: lustre-MDT0000: new disk, initializing [ 7818.806978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7818.814975] Lustre: Skipped 2 previous similar messages [ 7818.833195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7823.473278] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 7833.973788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7834.121442] Lustre: 218503:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7834.418898] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7838.722644] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 7848.404636] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7853.718187] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7860.591184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7860.927457] Lustre: 220472:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7863.074620] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7867.909474] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 7877.255839] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7881.512718] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7881.680785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7886.343613] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7886.602500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7888.876782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7888.878226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7888.903564] Lustre: Skipped 2 previous similar messages [ 7888.918569] LustreError: Skipped 2 previous similar messages [ 7888.919958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7888.919965] Lustre: Skipped 7 previous similar messages [ 7894.572829] Lustre: server umount lustre-OST0000 complete [ 7903.725926] Lustre: server umount lustre-MDT0000 complete [ 7904.225768] LustreError: 218509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7904.239076] LustreError: 218509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 7906.764538] LustreError: 218494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787891662 with bad export cookie 7815754045765522416 [ 7906.773032] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7906.778220] LustreError: 218494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7907.084911] Lustre: server umount lustre-MDT0001 complete [ 7914.962587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7919.962756] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 7927.445604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7931.903816] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 7938.288457] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7942.442327] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7947.783258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7952.614773] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 7958.063747] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7973.860104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7978.495314] Lustre: server umount lustre-OST0000 complete [ 7987.815745] Lustre: server umount lustre-MDT0000 complete [ 7991.063473] LustreError: 222670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787891746 with bad export cookie 7815754045765523305 [ 7991.067278] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7991.076876] LustreError: 222670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7991.381230] Lustre: server umount lustre-MDT0001 complete [ 8097.655233] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 8100.145316] Key type lgssc unregistered [ 8100.380520] LNet: 226373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8100.389755] LNetError: 226373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8100.402967] LNet: Removed LNI 192.168.203.141@tcp [ 8101.141165] Key type .llcrypt unregistered [ 8101.145051] Key type ._llcrypt unregistered [ 8126.271539] Key type ._llcrypt registered [ 8126.274780] Key type .llcrypt registered [ 8126.380120] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [ 8144.479820] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 8145.857832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8145.895118] alg: No test for adler32 (adler32-zlib) [ 8146.928685] Lustre: Lustre: Build Version: 2.17.57_84_gd387c42 [ 8147.255694] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 8148.976202] Key type lgssc registered [ 8150.388375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8181.104839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8181.140343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8182.410842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8182.446501] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8182.548901] Lustre: lustre-MDT0000: new disk, initializing [ 8182.670614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8182.715429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8186.739355] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 8195.366809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8195.502057] Lustre: 229721:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8195.551874] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8195.555649] Lustre: Skipped 1 previous similar message [ 8195.624282] Lustre: lustre-MDT0001: new disk, initializing [ 8195.734333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8195.773880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8195.789852] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8199.860651] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 8208.173158] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8212.977707] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8218.656942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8218.946299] Lustre: lustre-OST0000: new disk, initializing [ 8218.955717] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8218.962811] Lustre: 231688:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8219.039926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8224.440833] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 8226.367541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8226.379537] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8226.480218] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8232.777796] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8237.155435] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8237.344605] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8241.579520] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8241.779337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8247.268067] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8247.270205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8247.275932] LustreError: Skipped 1 previous similar message [ 8247.280460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8247.284678] Lustre: Skipped 1 previous similar message [ 8247.303142] Lustre: Skipped 1 previous similar message [ 8249.701568] Lustre: server umount lustre-OST0000 complete [ 8257.505480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8257.508475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8257.515667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8257.548102] Lustre: Skipped 1 previous similar message [ 8259.367034] Lustre: server umount lustre-MDT0000 complete [ 8263.108666] LustreError: 231687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787892018 with bad export cookie 11280943777940377574 [ 8263.110679] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8263.134368] LustreError: 231687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8263.579334] Lustre: server umount lustre-MDT0001 complete [ 8282.961746] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 8293.402684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8293.932548] LustreError: 234621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8293.962781] LustreError: 234621:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8293.999198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8299.490724] LustreError: 234622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8299.938382] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 8303.586708] LustreError: 234621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8308.705789] LustreError: 234622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8310.579395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8311.003316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8315.586113] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 8318.156781] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8323.660401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8323.962161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8329.300788] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 8337.805089] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8337.977175] Lustre: lustre-OST0001: new disk, initializing [ 8337.980499] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8337.985477] Lustre: 236929:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8344.642782] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 8346.739080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8346.747168] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8346.848474] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8356.923804] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8365.307910] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8379.046501] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 00:42:13 (1787892133) [ 8382.950421] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8382.952980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8382.954772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8382.965046] Lustre: Skipped 3 previous similar messages [ 8382.978367] Lustre: Skipped 4 previous similar messages [ 8388.068899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8388.076747] Lustre: Skipped 3 previous similar messages [ 8393.191610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8395.232497] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8395.399382] LustreError: 238059:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff8894d6c8c100) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 8395.488784] Lustre: server umount lustre-MDT0000 complete [ 8398.317499] LustreError: 234622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8398.349155] LustreError: 234622:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8398.981206] LustreError: 234603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787892154 with bad export cookie 11280943777940378456 [ 8398.984785] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8398.993751] LustreError: 234603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8399.286442] Lustre: server umount lustre-MDT0001 complete [ 8412.812412] Lustre: server umount lustre-OST0000 complete [ 8426.350434] Lustre: server umount lustre-OST0001 complete [ 8432.601241] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [ 8441.937413] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 8473.644683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8473.946929] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8473.976585] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8474.091801] Lustre: lustre-MDT0000: new disk, initializing [ 8474.213055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8474.219402] Lustre: Skipped 1 previous similar message [ 8474.258114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8478.502936] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 8489.580133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8489.778847] Lustre: 240876:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8489.843071] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8489.854913] Lustre: Skipped 1 previous similar message [ 8489.950205] Lustre: lustre-MDT0001: new disk, initializing [ 8490.111128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8490.133113] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8495.187903] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 8505.849776] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8511.631482] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8519.188667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8519.451728] Lustre: lustre-OST0000: new disk, initializing [ 8519.455619] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8519.462914] Lustre: 242846:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8519.592506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8519.600012] Lustre: Skipped 1 previous similar message [ 8520.874528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8520.892233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8520.990883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8525.070233] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 8532.702507] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8537.360607] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8537.644760] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8541.809859] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8542.034805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8546.787364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8546.791626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8546.822915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8546.854915] Lustre: Skipped 4 previous similar messages [ 8550.455929] Lustre: server umount lustre-OST0000 complete [ 8556.513406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8556.524552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8556.535259] Lustre: Skipped 1 previous similar message [ 8560.763551] Lustre: server umount lustre-MDT0000 complete [ 8562.149052] LustreError: 240886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8562.167437] LustreError: 240886:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8564.206649] LustreError: 240866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787892320 with bad export cookie 11280943777940380591 [ 8564.213210] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8564.217980] LustreError: 240866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8564.550917] Lustre: server umount lustre-MDT0001 complete [ 8571.916621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8576.583406] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 8582.630397] LustreError: 245062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8582.648399] LustreError: 245062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8584.096674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8584.489372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8584.501517] Lustre: Skipped 1 previous similar message [ 8588.758642] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 8595.667868] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8601.220967] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8607.363102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8613.363970] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 8621.793442] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8771.556204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8771.566158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8771.591897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8771.622442] Lustre: Skipped 2 previous similar messages [ 8771.647378] Lustre: Skipped 4 previous similar messages [ 8775.368439] Lustre: server umount lustre-OST0000 complete [ 8785.596838] Lustre: server umount lustre-MDT0000 complete [ 8786.913617] LustreError: 245057:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8786.956928] LustreError: 245057:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8789.657460] LustreError: 245042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787892545 with bad export cookie 11280943777940381473 [ 8789.667851] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8789.678394] LustreError: 245042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8790.079870] Lustre: server umount lustre-MDT0001 complete [ 8888.591386] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 8891.083912] Key type lgssc unregistered [ 8891.431342] LNet: 248679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8891.444218] LNetError: 248679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8891.465202] LNet: Removed LNI 192.168.203.141@tcp [ 8892.281214] Key type .llcrypt unregistered [ 8892.285741] Key type ._llcrypt unregistered [ 8917.363094] Key type ._llcrypt registered [ 8917.364834] Key type .llcrypt registered [ 8917.468916] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [ 8932.161585] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 8932.936494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8933.062768] alg: No test for adler32 (adler32-zlib) [ 8934.124937] Lustre: Lustre: Build Version: 2.17.57_84_gd387c42 [ 8934.368717] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 8936.016248] Key type lgssc registered [ 8937.104904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8976.204591] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 8984.924708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8984.938726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8986.185476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8986.224764] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8986.342393] Lustre: lustre-MDT0000: new disk, initializing [ 8986.420953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8986.439767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8990.030129] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 8999.110564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8999.257430] Lustre: 252744:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8999.313044] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8999.316841] Lustre: Skipped 1 previous similar message [ 8999.381223] Lustre: lustre-MDT0001: new disk, initializing [ 8999.521859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8999.574692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8999.592478] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9003.428605] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9008.128555] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9013.837673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9014.100251] Lustre: lustre-OST0000: new disk, initializing [ 9014.106584] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9014.117703] Lustre: 254378:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9014.204516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9016.830409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9016.850257] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9017.001460] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9019.490559] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9029.226877] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9029.389444] Lustre: lustre-OST0001: new disk, initializing [ 9029.394357] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9029.400734] Lustre: 255248:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9029.547301] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9035.969827] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9038.930499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9038.945606] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9039.100416] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9047.984989] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9059.796849] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9075.177792] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9075.189015] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9075.205502] Lustre: Skipped 1 previous similar message [ 9075.239791] Lustre: Skipped 3 previous similar messages [ 9080.294804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9080.307027] Lustre: Skipped 3 previous similar messages [ 9081.584403] Lustre: server umount lustre-MDT0000 complete [ 9085.724735] LustreError: 252738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787892841 with bad export cookie 14057091380387592517 [ 9085.725852] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9085.743287] LustreError: 252738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9086.015533] Lustre: server umount lustre-MDT0001 complete [ 9100.381883] Lustre: server umount lustre-OST0000 complete [ 9113.942641] Lustre: server umount lustre-OST0001 complete [ 9127.988501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9138.249552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9147.792044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9156.990707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9176.447103] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 9185.918462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9185.938162] Lustre: lustre-MDT0000: reset Object Index mappings [ 9186.266351] LustreError: 260262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9186.298498] LustreError: 260262:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9186.390439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9190.814640] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9191.393542] LustreError: 260263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9196.544024] LustreError: 260262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9199.160909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9199.181295] Lustre: lustre-MDT0001: reset Object Index mappings [ 9199.577194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9203.628247] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9206.626098] Lustre: 261405:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9212.655462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9219.523401] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9224.372084] LustreError: 261760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9227.767034] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9228.028037] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9228.033170] Lustre: Skipped 1 previous similar message [ 9232.125987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9232.141412] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9233.533547] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9239.999180] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9243.157831] Lustre: 263277:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9247.720727] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9247.722164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9247.731175] Lustre: Skipped 4 previous similar messages [ 9247.745432] Lustre: Skipped 3 previous similar messages [ 9249.874248] Lustre: server umount lustre-MDT0000 complete [ 9252.761329] LustreError: 260244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787893008 with bad export cookie 14057091380387599307 [ 9252.763693] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9252.774976] LustreError: 260244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9252.836018] LustreError: 262065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9252.836078] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9252.839367] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9252.857868] LustreError: 262065:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9252.865843] Lustre: Skipped 2 previous similar messages [ 9252.984596] Lustre: server umount lustre-MDT0001 complete [ 9262.281428] Lustre: server umount lustre-OST0000 complete [ 9265.293543] Lustre: server umount lustre-OST0001 complete [ 9281.529466] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 00:57:15 (1787893035) [ 9294.297394] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [ 9302.490672] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 9335.185436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9335.353358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9335.387548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9335.471928] Lustre: lustre-MDT0000: new disk, initializing [ 9335.539851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9335.558860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9338.827628] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9348.212423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9348.324224] Lustre: 267525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9348.379252] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9348.382963] Lustre: Skipped 1 previous similar message [ 9348.480952] Lustre: lustre-MDT0001: new disk, initializing [ 9348.651792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9348.677398] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9353.305959] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9362.509272] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9368.331676] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9374.580864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9374.989632] Lustre: lustre-OST0000: new disk, initializing [ 9375.003260] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9375.013734] Lustre: 269494:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9376.932080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9376.947765] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9377.076989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9380.976092] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9390.490487] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9395.164529] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9395.477705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9400.718153] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9400.919558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9402.855586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9402.856807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9402.873805] Lustre: Skipped 1 previous similar message [ 9402.884321] Lustre: Skipped 2 previous similar messages [ 9408.605379] Lustre: server umount lustre-OST0000 complete [ 9413.092588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9413.114061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9413.115409] Lustre: Skipped 1 previous similar message [ 9413.123892] Lustre: Skipped 2 previous similar messages [ 9426.400396] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9426.577239] Lustre: server umount lustre-MDT0000 complete [ 9428.449406] LustreError: 267538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9428.466385] LustreError: 267538:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9429.857429] LustreError: 267520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787893185 with bad export cookie 14057091380387601400 [ 9429.860498] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9429.870504] LustreError: 267520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9430.101793] Lustre: server umount lustre-MDT0001 complete [ 9446.556309] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 9455.725442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9456.070107] LustreError: 272429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9456.130277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9456.135240] Lustre: Skipped 2 previous similar messages [ 9460.174454] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9467.985465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9472.664924] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9475.830080] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9482.091367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9488.669113] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9497.066640] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9497.248961] Lustre: lustre-OST0001: new disk, initializing [ 9497.251779] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9497.266079] Lustre: 274737:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9498.594763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9498.610733] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9498.710802] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9503.500520] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9513.196890] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9516.827726] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9522.499277] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9522.536110] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9529.825387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9529.837763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9529.860996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9529.867971] Lustre: Skipped 5 previous similar messages [ 9535.548872] Lustre: server umount lustre-MDT0000 complete [ 9536.488826] LustreError: 272426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9536.514867] LustreError: 272426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9538.994301] LustreError: 272409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787893294 with bad export cookie 14057091380387602282 [ 9539.001148] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9539.009061] LustreError: 272409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9539.315725] Lustre: server umount lustre-MDT0001 complete [ 9552.375383] Lustre: server umount lustre-OST0000 complete [ 9566.429365] Lustre: server umount lustre-OST0001 complete [ 9581.571212] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 9591.180643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9591.609298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9591.615164] Lustre: Skipped 3 previous similar messages [ 9595.791461] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9602.021229] LustreError: 277807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9602.043430] LustreError: 277807:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9603.705092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9608.280498] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9610.749484] Lustre: 278946:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9617.131857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9624.568428] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9633.864658] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9637.189665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9637.251654] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9639.931147] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9646.714450] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9649.763631] Lustre: 280824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9654.464289] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9654.478973] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9662.745671] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 01:03:36 (1787893416) [ 9678.309453] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9678.312041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9678.322141] Lustre: Skipped 6 previous similar messages [ 9678.325073] Lustre: Skipped 6 previous similar messages [ 9683.795458] Lustre: server umount lustre-MDT0000 complete [ 9686.951249] LustreError: 277789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787893442 with bad export cookie 14057091380387608813 [ 9686.960743] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9686.972082] LustreError: 277789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9701.344128] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9701.532683] Lustre: server umount lustre-MDT0001 complete [ 9710.646391] Lustre: server umount lustre-OST0000 complete [ 9719.489708] Lustre: server umount lustre-OST0001 complete [ 9725.410166] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [ 9734.016863] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 9771.214291] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 9778.271881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9778.488956] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9778.519673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9778.575925] Lustre: lustre-MDT0000: new disk, initializing [ 9778.687353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9782.111636] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9793.481082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9793.737257] Lustre: 285060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9794.233371] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9798.482784] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9810.560214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9810.754430] Lustre: 286695:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9812.566930] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9817.038670] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9827.127493] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9827.414053] Lustre: 287566:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9828.860623] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9828.875867] Lustre: Skipped 1 previous similar message [ 9829.006328] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9833.335141] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9842.025628] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9845.478294] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9845.483254] Lustre: Skipped 1 previous similar message [ 9860.243727] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 01:06:53 (1787893613) [ 9864.577324] Lustre: server umount lustre-MDT0000 complete [ 9864.676373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9864.678408] LustreError: 285070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9864.699955] Lustre: Skipped 4 previous similar messages [ 9864.717085] LustreError: 285070:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 9867.461875] LustreError: 285053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787893623 with bad export cookie 14057091380387616464 [ 9867.469642] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9867.470706] LustreError: 285053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9867.776767] Lustre: server umount lustre-MDT0001 complete [ 9881.887256] Lustre: server umount lustre-OST0000 complete [ 9895.935505] Lustre: server umount lustre-OST0001 complete [ 9902.717601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9903.295131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9903.307280] Lustre: Skipped 7 previous similar messages [ 9907.228829] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9914.981706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9919.261043] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9925.849641] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9930.822877] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9936.056379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9941.178924] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9947.356870] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9962.555103] loop1: detected capacity change from 0 to 204800000 [ 9962.817900] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9962.842551] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9962.853030] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9967.513533] loop1: detected capacity change from 0 to 204800000 [ 9972.708755] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9972.816384] LustreError: 289803:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs24288: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 9972.830840] LustreError: 289803:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs24288 [ 9972.835851] LustreError: 289803:0:(mgs_handler.c:517:mgs_target_reg()) fs24288-MDT0001: Error checking target index: rc = -13 [ 9972.842936] LustreError: fs24288-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9972.850090] LustreError: 293530:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [ 9972.856277] LustreError: 293530:0:(tgt_mount.c:2132:server_put_super()) no obd fs24288-MDT0001 [ 9972.860883] LustreError: 293530:0:(tgt_mount.c:116:server_deregister_mount()) fs24288-MDT0001 not registered [ 9972.891590] Lustre: server umount fs24288-MDT0001 complete [ 9972.895442] LustreError: 293530:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9982.099295] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9982.227627] Lustre: Setting parameter fs24288-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs24288-MDT0001 [ 9982.261830] Lustre: srv-fs24288-MDT0001: No data found on store. Initialize space. [ 9982.265438] Lustre: Skipped 4 previous similar messages [ 9982.314975] Lustre: fs24288-MDT0001: new disk, initializing [ 9982.317870] Lustre: Skipped 3 previous similar messages [ 9982.447326] Lustre: srv-fs24288-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9982.463271] Lustre: 293997:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs24288-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9982.476656] LustreError: 293997:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs24288-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9986.529686] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 9992.205628] Lustre: server umount fs24288-MDT0001 complete [10002.864230] loop1: detected capacity change from 0 to 204800000 [10007.254479] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10007.493305] Lustre: 295565:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs24288-MDT0001-mdtlov: can't create new llog: rc = -115 [10007.505729] LustreError: 295565:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs24288-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10011.509968] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [10015.869576] Lustre: server umount fs24288-MDT0001 complete [10018.285071] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10023.398790] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10023.407610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10023.409926] Lustre: Skipped 3 previous similar messages [10023.417254] Lustre: Skipped 12 previous similar messages [10029.655851] Lustre: server umount lustre-MDT0000 complete [10035.685378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10035.793293] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10039.418325] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [10041.327921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10045.547871] loop1: detected capacity change from 0 to 204800000 [10050.504605] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10050.795242] Lustre: 298009:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs24288-MDT0001-mdtlov: can't create new llog: rc = -115 [10050.802893] LustreError: 298009:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs24288-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10054.654079] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [10058.786373] Lustre: server umount fs24288-MDT0001 complete [10082.291850] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10086.586592] Lustre: server umount lustre-OST0000 complete [10092.528943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10106.336142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10106.531507] Lustre: server umount lustre-MDT0000 complete [10110.774556] LustreError: 289797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787893866 with bad export cookie 14057091380387619145 [10110.777839] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10110.791474] LustreError: 289797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10111.333717] Lustre: server umount lustre-MDT0001 complete [10121.484831] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [10124.343503] Key type lgssc unregistered [10124.751905] LNet: 300353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10124.758107] LNetError: 300353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10124.783415] LNet: Removed LNI 192.168.203.141@tcp [10125.774209] Key type .llcrypt unregistered [10125.778762] Key type ._llcrypt unregistered [10141.629704] Key type ._llcrypt registered [10141.631668] Key type .llcrypt registered [10141.703397] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [10142.350828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10142.425538] alg: No test for adler32 (adler32-zlib) [10143.444259] Lustre: Lustre: Build Version: 2.17.57_84_gd387c42 [10143.629440] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [10145.296209] Key type lgssc registered [10146.161176] Lustre: Echo OBD driver; http://www.lustre.org/ [10151.904510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10151.942339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10153.319573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10156.809408] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [10163.138399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10163.393872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10166.816744] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [10172.174235] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10176.176200] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10181.296916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10181.542631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10186.489673] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [10188.714640] LustreError: 303430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10188.738181] LustreError: 303430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10193.892205] LustreError: 303907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10193.960630] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10199.015423] LustreError: 303428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10204.954138] LustreError: 303428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10204.973817] LustreError: 303428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10210.006594] LustreError: 303428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10210.045401] LustreError: 303428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10220.240117] LustreError: 303907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10220.257533] LustreError: 303907:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10222.988948] loop1: detected capacity change from 0 to 204800000 [10223.203383] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10223.214993] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10223.225514] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10228.121492] loop1: detected capacity change from 0 to 204800000 [10233.707352] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10233.840489] Lustre: fs24184-OST0000: new disk, initializing [10233.843864] Lustre: srv-fs24184-OST0000: No data found on store. Initialize space. [10233.849573] Lustre: 305358:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10233.914558] Lustre: fs24184-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10240.723736] LustreError: 303907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10240.765816] LustreError: 303907:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [10241.365829] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [10248.978047] Lustre: server umount fs24184-OST0000 complete [10259.655070] loop1: detected capacity change from 0 to 204800000 [10264.372941] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10264.587937] Lustre: fs24184-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10269.838850] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [10271.723232] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10276.566086] LustreError: 303431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10276.600809] LustreError: 303431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [10277.069704] loop2: detected capacity change from 0 to 204800000 [10277.278703] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10277.293478] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10277.310228] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10281.421364] loop2: detected capacity change from 0 to 204800000 [10281.965763] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10286.768764] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10286.840509] LustreError: fs24184-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10286.853814] LustreError: fs24184-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10286.862191] LustreError: 308398:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [10286.867909] LustreError: 308398:0:(tgt_mount.c:2132:server_put_super()) no obd fs24184-OST0001 [10286.873261] LustreError: 308398:0:(tgt_mount.c:116:server_deregister_mount()) fs24184-OST0001 not registered [10286.898325] Lustre: server umount fs24184-OST0001 complete [10286.901439] LustreError: 308398:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10296.614516] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10296.749826] Lustre: fs24184-OST0001: new disk, initializing [10296.757221] Lustre: srv-fs24184-OST0001: No data found on store. Initialize space. [10296.770671] Lustre: 308949:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10296.854638] Lustre: fs24184-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10302.997389] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [10311.102212] Lustre: server umount fs24184-OST0000 complete [10320.094192] Lustre: server umount fs24184-OST0001 complete [10331.763231] loop1: detected capacity change from 0 to 204800000 [10336.838769] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10336.991698] Lustre: fs24184-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10342.691799] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [10343.120108] LustreError: 303907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10343.152273] LustreError: 303907:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [10349.270761] loop2: detected capacity change from 0 to 204800000 [10353.988611] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10361.578780] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [10370.441402] Lustre: server umount fs24184-OST0000 complete [10378.100766] Lustre: server umount fs24184-OST0001 complete [10388.449391] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10388.460349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10388.469842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10389.476356] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10389.477459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10389.497968] Lustre: Skipped 1 previous similar message [10393.275549] Lustre: server umount lustre-OST0000 complete [10399.200873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10399.210760] LustreError: Skipped 1 previous similar message [10399.214346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10399.228324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10402.495500] Lustre: server umount lustre-MDT0000 complete [10406.229868] LustreError: 301582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787894162 with bad export cookie 10180953115308119610 [10406.232525] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10406.236296] LustreError: 301582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10407.009071] Lustre: server umount lustre-MDT0001 complete [10418.321430] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [10421.625602] Key type lgssc unregistered [10421.863608] LNet: 314872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10421.868857] LNetError: 314872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10421.878513] LNet: Removed LNI 192.168.203.141@tcp [10422.747398] Key type .llcrypt unregistered [10422.754403] Key type ._llcrypt unregistered [10449.943332] Key type ._llcrypt registered [10449.946376] Key type .llcrypt registered [10450.095539] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [10462.878122] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [10463.754217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10463.903600] alg: No test for adler32 (adler32-zlib) [10464.876295] Lustre: Lustre: Build Version: 2.17.57_84_gd387c42 [10465.146560] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [10466.784357] Key type lgssc registered [10467.797544] Lustre: Echo OBD driver; http://www.lustre.org/ [10498.522747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10498.552385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10499.753098] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10499.780273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10499.834388] Lustre: lustre-MDT0000: new disk, initializing [10499.888486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10499.905298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10503.399583] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [10512.425068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10512.591933] Lustre: 318708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10512.658830] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10512.662819] Lustre: Skipped 1 previous similar message [10512.738962] Lustre: lustre-MDT0001: new disk, initializing [10512.861512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10512.910600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10512.922649] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10516.819265] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [10526.832608] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10533.291857] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10540.417316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10540.714543] Lustre: lustre-OST0000: new disk, initializing [10540.718596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10540.727340] Lustre: 320676:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10540.849095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10546.844427] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [10549.839761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10549.855915] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10549.981054] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10555.124743] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10556.872977] Lustre: Mounted lustre-client - version 2.17.57_84_gd387c42 [10557.776158] Lustre: Unmounted lustre-client [10567.144035] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10567.151399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10567.169130] Lustre: Skipped 1 previous similar message [10572.101632] Lustre: server umount lustre-OST0000 complete [10577.379684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10577.397135] Lustre: Skipped 1 previous similar message [10577.402855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10577.416152] Lustre: Skipped 2 previous similar messages [10582.498864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10582.506438] Lustre: Skipped 1 previous similar message [10587.618312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10587.625657] Lustre: Skipped 1 previous similar message [10590.176482] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10590.406024] Lustre: server umount lustre-MDT0000 complete [10594.048742] LustreError: 322084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787894349 with bad export cookie 11360821333192045488 [10594.054288] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10594.067874] LustreError: 322084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10594.608597] Lustre: server umount lustre-MDT0001 complete [10606.082254] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [10608.719848] Key type lgssc unregistered [10609.028881] LNet: 322838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10609.040825] LNetError: 322838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10687.910767] LNet: Removed LNI 192.168.203.141@tcp [10688.596584] Key type .llcrypt unregistered [10688.601537] Key type ._llcrypt unregistered [10704.950422] Key type ._llcrypt registered [10704.954348] Key type .llcrypt registered [10705.062942] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [10720.674434] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [10721.837795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10721.922400] alg: No test for adler32 (adler32-zlib) [10722.996985] Lustre: Lustre: Build Version: 2.17.57_84_gd387c42 [10723.243383] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [10724.904263] Key type lgssc registered [10725.880708] Lustre: Echo OBD driver; http://www.lustre.org/ [10752.233455] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 01:21:46 (1787894506) [10758.132233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10758.161067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10759.384843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10759.418486] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10759.488278] Lustre: lustre-MDT0000: new disk, initializing [10759.549849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10759.571352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10762.793456] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [10771.796488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10771.914631] Lustre: 326051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10771.952254] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10771.956651] Lustre: Skipped 1 previous similar message [10772.041625] Lustre: lustre-MDT0001: new disk, initializing [10772.127898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10772.167403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10772.185899] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10776.878670] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [10785.718841] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10790.687191] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10796.704628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10796.889752] Lustre: lustre-OST0000: new disk, initializing [10796.893902] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10796.900431] Lustre: 328023:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10796.992305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10802.139965] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [10804.350932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10804.368312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10804.467610] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10810.843601] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10825.186573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10825.191053] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10825.191372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10825.198790] Lustre: Skipped 1 previous similar message [10830.309460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10830.319114] Lustre: Skipped 2 previous similar messages [10831.259669] Lustre: server umount lustre-OST0000 complete [10835.425653] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10835.447404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10835.457666] Lustre: Skipped 1 previous similar message [10840.546029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10840.550978] Lustre: Skipped 1 previous similar message [10840.898249] Lustre: server umount lustre-MDT0000 complete [10845.011408] LustreError: 329181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787894600 with bad export cookie 5753748783925009278 [10845.023567] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10845.026748] LustreError: 329181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10845.263128] Lustre: server umount lustre-MDT0001 complete [10856.938858] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [10859.328415] Key type lgssc unregistered [10859.561378] LNet: 329931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10859.568531] LNetError: 329931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10859.586907] LNet: Removed LNI 192.168.203.141@tcp [10860.444980] Key type .llcrypt unregistered [10860.447460] Key type ._llcrypt unregistered [10881.089421] Key type ._llcrypt registered [10881.094122] Key type .llcrypt registered [10881.209853] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [10882.310297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10882.388277] alg: No test for adler32 (adler32-zlib) [10883.465653] Lustre: Lustre: Build Version: 2.17.57_84_gd387c42 [10883.727683] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [10885.400686] Key type lgssc registered [10886.588898] Lustre: Echo OBD driver; http://www.lustre.org/ [10893.789080] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10893.814296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10895.276047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10898.836777] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [10905.577311] LustreError: 331204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10905.591864] LustreError: 331204:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10906.777264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10907.249694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10911.488381] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [10918.485440] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10923.676588] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10930.540382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10930.958991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10936.751881] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [10943.534957] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10953.595764] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10955.019422] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10958.307734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10958.315725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10958.332178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10961.650846] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.41@tcp (stopping) [10961.897809] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10961.913844] Lustre: Skipped 1 previous similar message [10964.639405] Lustre: server umount lustre-MDT0000 complete [10966.734160] LustreError: 331200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10968.699100] LustreError: 331185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787894724 with bad export cookie 8564827270266970657 [10968.705232] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10968.712595] LustreError: 331185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10971.852870] LustreError: 332257:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10971.868706] LustreError: 332257:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10971.897788] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.41@tcp (stopping) [10971.909050] Lustre: Skipped 3 previous similar messages [10972.132231] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10976.975808] LustreError: 331200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10976.976542] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.41@tcp (stopping) [10976.992818] LustreError: 331200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10977.002117] Lustre: Skipped 2 previous similar messages [10982.092715] LustreError: 331198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10982.104202] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.41@tcp (stopping) [10982.114042] LustreError: 331198:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10982.134457] Lustre: Skipped 1 previous similar message [10983.392632] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10983.626488] Lustre: server umount lustre-MDT0001 complete [10987.297410] Lustre: server umount lustre-OST0000 complete [10995.459870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11000.000983] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [11003.140283] Lustre: 335384:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.141@tcp [11005.721866] Lustre: server umount MGS complete [11013.654181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11014.347906] LustreError: 335956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11014.367800] LustreError: 335956:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11014.436306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11018.509670] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [11026.614276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11026.901232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11030.970644] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [11039.008636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11039.204591] LustreError: 337390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11039.226523] LustreError: 337390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11039.351235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11045.366381] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [11052.528625] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11054.239985] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11059.618414] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11061.040520] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11066.852730] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11066.862227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11066.880374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11066.892106] Lustre: Skipped 2 previous similar messages [11067.592628] Lustre: server umount lustre-OST0000 complete [11072.482209] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11072.489204] Lustre: Skipped 1 previous similar message [11077.809516] Lustre: server umount lustre-MDT0000 complete [11081.522610] LustreError: 335939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787894837 with bad export cookie 8564827270266971497 [11081.533990] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11081.546336] LustreError: 335939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11081.871723] Lustre: server umount lustre-MDT0001 complete [11091.550173] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [11093.818708] Key type lgssc unregistered [11094.045750] LNet: 339390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11094.053182] LNetError: 339390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11094.067470] LNet: Removed LNI 192.168.203.141@tcp [11094.974376] Key type .llcrypt unregistered [11094.979270] Key type ._llcrypt unregistered [11111.342355] Key type ._llcrypt registered [11111.351200] Key type .llcrypt registered [11111.463683] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [11125.746701] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [11127.265440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11127.308053] alg: No test for adler32 (adler32-zlib) [11128.456989] Lustre: Lustre: Build Version: 2.17.57_84_gd387c42 [11128.707883] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [11130.384925] Key type lgssc registered [11131.544264] Lustre: Echo OBD driver; http://www.lustre.org/ [11164.335810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11164.352754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11165.542766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11165.576663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11165.645874] Lustre: lustre-MDT0000: new disk, initializing [11165.711634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11165.734979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11169.549405] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [11178.984079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11179.109778] Lustre: 342545:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11179.151926] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11179.156865] Lustre: Skipped 1 previous similar message [11179.231618] Lustre: lustre-MDT0001: new disk, initializing [11179.363970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11179.414600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11179.423120] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11184.351836] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [11195.996739] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11202.957671] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11209.675230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11210.093897] Lustre: lustre-OST0000: new disk, initializing [11210.101230] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11210.113276] Lustre: 344514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11210.247631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11217.049877] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [11217.540822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11217.565911] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11217.682259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11227.409983] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11232.560558] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11232.780705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11237.164527] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11237.354671] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11243.509860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11243.509890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11243.518632] LustreError: Skipped 1 previous similar message [11243.520221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11243.544709] Lustre: Skipped 1 previous similar message [11245.513975] Lustre: server umount lustre-OST0000 complete [11251.172469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11251.180917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11251.210756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11251.217983] Lustre: Skipped 1 previous similar message [11253.729930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11253.732520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11256.205045] Lustre: server umount lustre-MDT0000 complete [11259.594305] LustreError: 342538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787895015 with bad export cookie 4943495016398734727 [11259.594757] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11259.604466] LustreError: 342538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11259.871618] Lustre: server umount lustre-MDT0001 complete [11268.038969] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 01:30:21 (1787895021) [11284.242151] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [11294.349920] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [11330.467898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11330.703982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11330.736938] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11330.822386] Lustre: lustre-MDT0000: new disk, initializing [11330.891177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11330.905880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11334.759782] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [11346.214161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11346.341890] Lustre: 349273:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11346.425173] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11346.430118] Lustre: Skipped 1 previous similar message [11346.448876] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11346.457930] Lustre: Skipped 1 previous similar message [11346.542911] Lustre: lustre-MDT0001: new disk, initializing [11346.647553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11346.679876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11346.693871] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11350.848637] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [11360.188174] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11367.276795] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11375.022938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11375.378208] Lustre: lustre-OST0000: new disk, initializing [11375.381929] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11375.387731] Lustre: 351240:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11375.580054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11382.884365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11382.900131] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11383.051954] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11383.976902] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [11394.494990] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11434.483423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11434.486933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11434.496431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11434.511591] Lustre: Skipped 1 previous similar message [11434.524774] Lustre: Skipped 1 previous similar message [11437.928720] Lustre: server umount lustre-OST0000 complete [11444.193605] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11444.196685] LustreError: Skipped 1 previous similar message [11444.201369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11444.213205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11447.897155] Lustre: server umount lustre-MDT0000 complete [11449.825377] LustreError: 349284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11449.858119] LustreError: 349284:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11452.649929] LustreError: 349265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787895208 with bad export cookie 4943495016398735609 [11452.669572] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11452.674229] LustreError: 349265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11453.021397] Lustre: server umount lustre-MDT0001 complete [11463.824548] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [11467.489751] Key type lgssc unregistered [11467.874654] LNet: 353416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11467.888647] LNetError: 353416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11467.931256] LNet: Removed LNI 192.168.203.141@tcp [11468.802262] Key type .llcrypt unregistered [11468.804706] Key type ._llcrypt unregistered [11564.437390] Key type ._llcrypt registered [11564.442854] Key type .llcrypt registered [11564.629852] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [11564.973906] Key type .llcrypt unregistered [11564.976451] Key type ._llcrypt unregistered [11638.087215] Key type ._llcrypt registered [11638.090362] Key type .llcrypt registered [11638.224181] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [11638.524378] Key type .llcrypt unregistered [11638.526578] Key type ._llcrypt unregistered [11667.967260] Key type ._llcrypt registered [11667.969792] Key type .llcrypt registered [11668.061267] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [11668.352813] Key type .llcrypt unregistered [11668.358803] Key type ._llcrypt unregistered [11702.190929] Key type ._llcrypt registered [11702.194334] Key type .llcrypt registered [11702.282656] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [11702.534412] Key type .llcrypt unregistered [11702.539256] Key type ._llcrypt unregistered [11780.571618] Key type ._llcrypt registered [11780.574328] Key type .llcrypt registered [11780.715652] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [11781.049433] Key type .llcrypt unregistered [11781.052894] Key type ._llcrypt unregistered [11810.637168] Key type ._llcrypt registered [11810.639524] Key type .llcrypt registered [11810.719811] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [11810.942491] Key type .llcrypt unregistered [11810.949688] Key type ._llcrypt unregistered [11843.125076] Key type ._llcrypt registered [11843.126899] Key type .llcrypt registered [11843.259383] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [11843.522996] Key type .llcrypt unregistered [11843.527255] Key type ._llcrypt unregistered [11875.092371] Key type ._llcrypt registered [11875.096027] Key type .llcrypt registered [11875.213129] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [11875.516406] Key type .llcrypt unregistered [11875.521822] Key type ._llcrypt unregistered [11905.018370] Key type ._llcrypt registered [11905.022655] Key type .llcrypt registered [11905.151720] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [11905.412590] Key type .llcrypt unregistered [11905.416152] Key type ._llcrypt unregistered [11934.917783] Key type ._llcrypt registered [11934.922039] Key type .llcrypt registered [11935.024418] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [11935.248450] Key type .llcrypt unregistered [11935.253783] Key type ._llcrypt unregistered [11997.868684] Key type ._llcrypt registered [11997.872464] Key type .llcrypt registered [11998.054742] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [11998.683464] Key type .llcrypt unregistered [11998.689929] Key type ._llcrypt unregistered [12029.458283] Key type ._llcrypt registered [12029.461563] Key type .llcrypt registered [12029.572359] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [12029.945441] Key type .llcrypt unregistered [12029.951306] Key type ._llcrypt unregistered [12059.489649] Key type ._llcrypt registered [12059.491574] Key type .llcrypt registered [12059.565766] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [12059.805711] Key type .llcrypt unregistered [12059.808432] Key type ._llcrypt unregistered [12112.026591] Key type ._llcrypt registered [12112.028947] Key type .llcrypt registered [12112.112176] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [12127.676846] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [12129.037613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12129.077372] alg: No test for adler32 (adler32-zlib) [12130.249766] Lustre: Lustre: Build Version: 2.17.57_84_gd387c42 [12130.511560] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [12132.218245] Key type lgssc registered [12133.409829] Lustre: Echo OBD driver; http://www.lustre.org/ [12165.280907] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12165.324504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12166.632173] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12166.660602] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12166.728215] Lustre: tst250fs-MDT0000: new disk, initializing [12166.796114] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12166.815425] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12172.273418] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [12183.446708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12183.570895] Lustre: 373689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12183.665902] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12183.671915] Lustre: Skipped 1 previous similar message [12183.787643] Lustre: tst250fs-MDT0001: new disk, initializing [12183.924928] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12183.992250] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12184.008086] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12188.107517] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [12192.711464] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12199.161996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12199.418615] Lustre: tst250fs-OST0000: new disk, initializing [12199.425181] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12199.435088] Lustre: 375322:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12199.584580] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12205.085239] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12205.094094] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12205.128906] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [12205.391110] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [12216.510872] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12216.785346] Lustre: tst250fs-OST0001: new disk, initializing [12216.801907] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12216.813930] Lustre: 376194:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12216.963121] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12217.652626] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12217.676519] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12217.872837] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12224.846387] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [12232.288769] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12234.653600] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [12234.666098] Lustre: Skipped 1 previous similar message [12255.716149] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12255.737248] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12255.780047] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12258.786298] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12258.791612] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12258.802104] Lustre: Skipped 2 previous similar messages [12258.815318] Lustre: Skipped 3 previous similar messages [12261.101562] Lustre: server umount tst250fs-MDT0000 complete [12264.782511] LustreError: 373683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787896020 with bad export cookie 3762971070512811289 [12264.783820] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12264.793338] LustreError: 373683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12265.179247] Lustre: server umount tst250fs-MDT0001 complete [12278.413750] Lustre: server umount tst250fs-OST0000 complete [12293.202701] Lustre: server umount tst250fs-OST0001 complete [12313.668699] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [12321.474228] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [12349.615076] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 01:48:23 (1787896103) [12351.236199] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12365.933573] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [12374.856180] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [12407.786689] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12118 sec ========= 01:49:21 (1787896161) [12409.406780] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:49:23 (1787896163) === [12412.563635] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:49:26 (1787896166) === [12450.044971] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [12452.653837] Key type lgssc unregistered [12452.931309] LNet: 385320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12452.947481] LNetError: 385320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12452.970739] LNet: Removed LNI 192.168.203.141@tcp [12453.876844] Key type .llcrypt unregistered [12453.883295] Key type ._llcrypt unregistered