[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 452771242 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003243] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008703] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011135] LSM: Security Framework initializing [ 0.012053] Yama: becoming mindful. [ 0.013043] SELinux: Initializing. [ 0.014085] *** VALIDATE selinux *** [ 0.023009] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028859] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029160] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030119] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031107] *** VALIDATE tmpfs *** [ 0.033514] *** VALIDATE proc *** [ 0.035162] *** VALIDATE cgroup *** [ 0.036011] *** VALIDATE cgroup2 *** [ 0.038280] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039174] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041037] Spectre V2 : User space: Vulnerable [ 0.042024] Speculative Store Bypass: Vulnerable [ 0.044932] debug: unmapping init [mem 0xffffffffbe059000-0xffffffffbe060fff] [ 0.046945] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047788] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048024] ... version: 2 [ 0.049015] ... bit width: 48 [ 0.050013] ... generic registers: 4 [ 0.051016] ... value mask: 0000ffffffffffff [ 0.052014] ... max period: 00007fffffffffff [ 0.053016] ... fixed-purpose events: 3 [ 0.054014] ... event mask: 000000070000000f [ 0.055356] rcu: Hierarchical SRCU implementation. [ 0.057584] smp: Bringing up secondary CPUs ... [ 0.058606] x86: Booting SMP configuration: [ 0.059028] .... node #0, CPUs: #1 #2 #3 [ 0.062375] smp: Brought up 1 node, 4 CPUs [ 0.064010] smpboot: Max logical packages: 1 [ 0.064983] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141539] node 0 deferred pages initialised in 76ms [ 0.145128] devtmpfs: initialized [ 0.146342] x86/mm: Memory block size: 128MB [ 0.149962] gcov: version magic: 0x41383552 [ 0.153248] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155000] pinctrl core: initialized pinctrl subsystem [ 0.156172] [ 0.156464] ************************************************************* [ 0.158009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160008] ** ** [ 0.161008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.163012] ** ** [ 0.165009] ** This means that this kernel is built to expose internal ** [ 0.166007] ** IOMMU data structures, which may compromise security on ** [ 0.167007] ** your system. ** [ 0.169010] ** ** [ 0.172013] ** If you see this message and you are not debugging the ** [ 0.174011] ** kernel, report this immediately to your vendor! ** [ 0.177014] ** ** [ 0.179010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182015] ************************************************************* [ 0.184775] NET: Registered protocol family 16 [ 0.186422] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.189029] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.191077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.194106] cpuidle: using governor menu [ 0.195686] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.198387] PCI: Using configuration type 1 for base access [ 0.200137] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.207126] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.208020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.210153] cryptd: max_cpu_qlen set to 1000 [ 0.213172] ACPI: Added _OSI(Module Device) [ 0.214012] ACPI: Added _OSI(Processor Device) [ 0.216041] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.217010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.221690] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.228617] ACPI: Interpreter enabled [ 0.229060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.230014] ACPI: Using IOAPIC for interrupt routing [ 0.232094] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.235331] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.244405] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.247040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.249017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.253079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.258400] acpiphp: Slot [2] registered [ 0.259112] acpiphp: Slot [5] registered [ 0.261114] acpiphp: Slot [6] registered [ 0.263129] acpiphp: Slot [7] registered [ 0.264128] acpiphp: Slot [8] registered [ 0.266146] acpiphp: Slot [9] registered [ 0.267145] acpiphp: Slot [10] registered [ 0.269079] acpiphp: Slot [3] registered [ 0.270131] acpiphp: Slot [4] registered [ 0.271112] acpiphp: Slot [11] registered [ 0.272089] acpiphp: Slot [12] registered [ 0.273072] acpiphp: Slot [13] registered [ 0.274090] acpiphp: Slot [14] registered [ 0.275080] acpiphp: Slot [15] registered [ 0.276121] acpiphp: Slot [16] registered [ 0.278090] acpiphp: Slot [17] registered [ 0.279075] acpiphp: Slot [18] registered [ 0.280089] acpiphp: Slot [19] registered [ 0.281078] acpiphp: Slot [20] registered [ 0.282096] acpiphp: Slot [21] registered [ 0.283079] acpiphp: Slot [22] registered [ 0.284109] acpiphp: Slot [23] registered [ 0.285079] acpiphp: Slot [24] registered [ 0.287123] acpiphp: Slot [25] registered [ 0.288087] acpiphp: Slot [26] registered [ 0.289089] acpiphp: Slot [27] registered [ 0.291082] acpiphp: Slot [28] registered [ 0.292091] acpiphp: Slot [29] registered [ 0.293085] acpiphp: Slot [30] registered [ 0.294080] acpiphp: Slot [31] registered [ 0.295056] PCI host bridge to bus 0000:00 [ 0.297028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.298034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.300023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.302025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.304022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.306023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.307274] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.309970] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.313218] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.321740] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.327131] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.329018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.330022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.332018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.334621] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.338889] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.341039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.344904] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.350016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.364023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.370816] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.375516] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.385017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.394017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.416015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.425756] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.437021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.447022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.472019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.483959] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.491018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.499020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.514018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.524000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.533020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.543022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.556026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.563040] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.574019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.584021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.604021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.613025] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.619022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.624017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.634016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.642121] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.643232] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.644272] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.645189] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.647226] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.652062] iommu: Default domain type: Passthrough [ 0.653000] SCSI subsystem initialized [ 0.654181] ACPI: bus type USB registered [ 0.656111] usbcore: registered new interface driver usbfs [ 0.659126] usbcore: registered new interface driver hub [ 0.661163] usbcore: registered new device driver usb [ 0.664204] pps_core: LinuxPPS API ver. 1 registered [ 0.666011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.671067] PTP clock support registered [ 0.674010] EDAC MC: Ver: 3.0.0 [ 0.676181] PCI: Using ACPI for IRQ routing [ 0.677659] NetLabel: Initializing [ 0.679014] NetLabel: domain hash size = 128 [ 0.682013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.685111] NetLabel: unlabeled traffic allowed by default [ 0.687136] vgaarb: loaded [ 0.689246] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.690014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.697000] clocksource: Switched to clocksource kvm-clock [ 0.791741] VFS: Disk quotas dquot_6.6.0 [ 0.793375] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.795632] *** VALIDATE ramfs *** [ 0.796960] *** VALIDATE hugetlbfs *** [ 0.798645] pnp: PnP ACPI init [ 0.801140] pnp: PnP ACPI: found 6 devices [ 0.817217] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.820630] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.822896] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.825289] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.827949] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.830511] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.833177] NET: Registered protocol family 2 [ 0.835538] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.840470] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.844387] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.849523] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.853038] TCP: Hash tables configured (established 65536 bind 65536) [ 0.855541] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.858187] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.860874] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.863742] NET: Registered protocol family 1 [ 0.866786] RPC: Registered named UNIX socket transport module. [ 0.869168] RPC: Registered udp transport module. [ 0.872937] RPC: Registered tcp transport module. [ 0.874356] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.876493] NET: Registered protocol family 44 [ 0.877848] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.880211] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.882429] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.884528] PCI: CLS 0 bytes, default 64 [ 0.886278] Unpacking initramfs... [ 2.294612] debug: unmapping init [mem 0xffff95e73cc54000-0xffff95e73ffbffff] [ 2.298575] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.301136] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.304389] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.779307] Initialise system trusted keyrings [ 2.781455] Key type blacklist registered [ 2.783585] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.791848] zbud: loaded [ 2.794782] *** VALIDATE nfs *** [ 2.795863] *** VALIDATE nfs4 *** [ 2.797142] pstore: using deflate compression [ 2.799789] Platform Keyring initialized [ 2.927578] NET: Registered protocol family 38 [ 2.929900] Key type asymmetric registered [ 2.932158] Asymmetric key parser 'x509' registered [ 2.934166] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.937202] io scheduler mq-deadline registered [ 2.938466] io scheduler kyber registered [ 2.939700] io scheduler bfq registered [ 2.942175] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.944915] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.948320] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.951722] ACPI: Power Button [PWRF] [ 2.957387] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.963769] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.974412] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.979318] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.989915] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.016915] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.044029] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.049157] Non-volatile memory driver v1.3 [ 3.050839] Linux agpgart interface v0.103 [ 3.081694] virtio_blk virtio1: [vda] 149376 512-byte logical blocks (76.5 MB/72.9 MiB) [ 3.084765] vda: detected capacity change from 0 to 76480512 [ 3.101308] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.104199] vdb: detected capacity change from 0 to 1073741824 [ 3.118491] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.121406] vdc: detected capacity change from 0 to 2621440000 [ 3.141649] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.144691] vdd: detected capacity change from 0 to 2621440000 [ 3.160982] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.163560] vde: detected capacity change from 0 to 4294967296 [ 3.179240] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.181845] vdf: detected capacity change from 0 to 4294967296 [ 3.187449] libphy: Fixed MDIO Bus: probed [ 3.192442] usbcore: registered new interface driver usbserial_generic [ 3.195081] usbserial: USB Serial support registered for generic [ 3.196852] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.200477] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.201963] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.204608] mousedev: PS/2 mouse device common for all mice [ 3.207194] rtc_cmos 00:05: RTC can wake from S4 [ 3.209313] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.210656] rtc_cmos 00:05: registered as rtc0 [ 3.214817] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.217444] intel_pstate: CPU model not supported [ 3.221201] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.223926] hid: raw HID events driver (C) Jiri Kosina [ 3.227036] usbcore: registered new interface driver usbhid [ 3.227199] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.228687] usbhid: USB HID core driver [ 3.228903] drop_monitor: Initializing network drop monitor service [ 3.235345] Initializing XFRM netlink socket [ 3.237186] NET: Registered protocol family 10 [ 3.240325] Segment Routing with IPv6 [ 3.241488] NET: Registered protocol family 17 [ 3.243646] mpls_gso: MPLS GSO support [ 3.250573] RAS: Correctable Errors collector initialized. [ 3.252425] AVX version of gcm_enc/dec engaged. [ 3.253559] AES CTR mode by8 optimization enabled [ 3.330077] sched_clock: Marking stable (3330049134, 0)->(4239373601, -909324467) [ 3.334136] registered taskstats version 1 [ 3.336327] Loading compiled-in X.509 certificates [ 3.338668] zswap: loaded using pool lzo/zbud [ 3.361933] Key type big_key registered [ 3.375667] Key type encrypted registered [ 3.377584] ima: No TPM chip found, activating TPM-bypass! [ 3.380289] ima: Allocated hash algorithm: sha1 [ 3.382197] ima: No architecture policies found [ 3.384119] evm: Initialising EVM extended attributes: [ 3.386261] evm: security.selinux [ 3.387579] evm: security.ima [ 3.388751] evm: security.capability [ 3.390293] evm: HMAC attrs: 0x1 [ 3.392810] rtc_cmos 00:05: setting system clock to 2026-08-22 10:15:23 UTC (1787393723) [ 3.398909] debug: unmapping init [mem 0xffffffffbf003000-0xffffffffbf1fffff] [ 3.401136] debug: unmapping init [mem 0xffffffffbdd82000-0xffffffffbe058fff] [ 3.409130] Write protecting the kernel read-only data: 28672k [ 3.411410] debug: unmapping init [mem 0xffffffffbc403000-0xffffffffbc5fffff] [ 3.413451] debug: unmapping init [mem 0xffffffffbcd14000-0xffffffffbcdfffff] [ 3.447614] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.455775] systemd[1]: Detected virtualization kvm. [ 3.457289] systemd[1]: Detected architecture x86-64. [ 3.458518] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.482881] systemd[1]: No hostname configured. [ 3.484507] systemd[1]: Set hostname to . [ 3.486381] random: systemd: uninitialized urandom read (16 bytes read) [ 3.487926] systemd[1]: Initializing machine ID from random generator. [ 3.604975] random: systemd: uninitialized urandom read (16 bytes read) [ 3.607230] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.610719] random: systemd: uninitialized urandom read (16 bytes read) [ 3.613111] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.616781] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.243800] device-mapper: uevent: version 1.0.3 [ 4.245690] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.973423] virtio_net virtio0 ens2: renamed from eth0 [ 4.983439] random: fast init done [ 4.995779] scsi host0: ata_piix [ 5.053934] scsi host1: ata_piix [ 5.058913] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.061608] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.456604] dracut-initqueue[592]: RTNETLINK answers: File exists [ 9.896259] random: crng init done [ 9.897655] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.313601] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.496635] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.718732] SELinux: Disabled at runtime. [ 11.783667] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.793434] systemd[1]: Detected virtualization kvm. [ 11.795421] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.289703] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.294386] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.300101] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.305346] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.309082] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.315769] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.320507] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ 12.448282] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.769231] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.104375] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.227098] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.426244] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.544134] EDAC sbridge: Ver: 1.1.2 [ 17.184388] Key type dns_resolver registered [ 17.871665] NFS: Registering the id_resolver key type [ 17.875533] Key type id_resolver registered [ 17.879288] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (5s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg114-server login: [ 72.682775] hrtimer: interrupt took 6208667 ns [ 72.914873] libcfs: loading out-of-tree module taints kernel. [ 72.960383] Key type ._llcrypt registered [ 72.962560] Key type .llcrypt registered [ 73.083590] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 90.314988] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 91.891404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 91.909360] alg: No test for adler32 (adler32-zlib) [ 93.288896] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 94.179723] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 95.991640] Key type lgssc registered [ 97.623985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 114.029791] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 150.019456] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 163.748926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 163.784537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 165.146241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 165.193725] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 165.277380] Lustre: lustre-MDT0000: new disk, initializing [ 165.365073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 165.385486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 169.559363] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 182.386146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.497498] Lustre: 6508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 182.526378] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 182.532883] Lustre: Skipped 1 previous similar message [ 182.625177] Lustre: lustre-MDT0001: new disk, initializing [ 182.682287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 182.708639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 182.724802] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 187.137988] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.438163] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 200.310411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 200.539971] Lustre: lustre-OST0000: new disk, initializing [ 200.543345] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 200.547830] Lustre: 8444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 200.609304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 205.871471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 205.881896] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 205.978666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 206.714290] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.228590] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.414985] Lustre: lustre-OST0001: new disk, initializing [ 220.418518] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 220.423784] Lustre: 9516:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 220.523171] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 226.356498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 226.364719] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 226.451841] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 227.155996] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.291253] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 244.722526] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 250.652515] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing check_logdir /tmp/testlogs/ [ 255.022948] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing yml_node [ 259.300271] Lustre: DEBUG MARKER: Client: 2.17.54.107 [ 261.904286] Lustre: DEBUG MARKER: MDS: 2.17.54.107 [ 264.178935] Lustre: DEBUG MARKER: OSS: 2.17.54.107 [ 265.585474] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 06:19:45 EDT 2026 [ 280.947938] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 282.678624] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 298.473523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 298.493413] Lustre: Skipped 2 previous similar messages [ 298.493546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 298.509175] Lustre: Skipped 2 previous similar messages [ 303.604100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 303.618039] Lustre: Skipped 4 previous similar messages [ 304.488746] Lustre: server umount lustre-MDT0000 complete [ 308.403995] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394028 with bad export cookie 1694079264662160823 [ 308.415984] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 308.425851] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 308.709630] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 308.722182] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 308.724080] Lustre: Skipped 2 previous similar messages [ 308.744890] Lustre: Skipped 1 previous similar message [ 313.827265] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 313.830575] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 313.852221] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 314.789217] Lustre: server umount lustre-MDT0001 complete [ 318.070496] Lustre: server umount lustre-OST0000 complete [ 321.203443] Lustre: server umount lustre-OST0001 complete [ 328.417693] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 340.078703] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 374.527549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 374.685867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 374.710788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 374.798152] Lustre: lustre-MDT0000: new disk, initializing [ 374.872930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 374.885344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 378.925153] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 388.286583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 388.408990] Lustre: 15070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 388.480562] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 388.484990] Lustre: Skipped 1 previous similar message [ 388.585644] Lustre: lustre-MDT0001: new disk, initializing [ 388.711369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 388.747142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 388.762544] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 393.119648] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 402.304620] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 408.107594] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 414.038853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 414.197654] Lustre: lustre-OST0000: new disk, initializing [ 414.200819] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 414.205638] Lustre: 17038:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 414.269379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 415.917136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 415.928213] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 416.021516] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 418.226967] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 424.486288] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 427.425810] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 427.581718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 430.061132] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 430.153786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 431.587390] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 431.587527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 431.593921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 431.600528] Lustre: Skipped 1 previous similar message [ 431.608513] Lustre: Skipped 2 previous similar messages [ 437.401531] Lustre: server umount lustre-OST0000 complete [ 441.827802] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 441.831732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 441.848635] Lustre: Skipped 1 previous similar message [ 441.852517] Lustre: Skipped 2 previous similar messages [ 446.962079] LustreError: 15081:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 446.977397] LustreError: 15081:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 447.053846] Lustre: server umount lustre-MDT0000 complete [ 450.458535] LustreError: 15866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394170 with bad export cookie 1694079264662162356 [ 450.461085] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 450.475158] LustreError: 15866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 450.869041] Lustre: server umount lustre-MDT0001 complete [ 456.364585] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 06:22:55 (1787394175) [ 473.662824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 474.062533] LustreError: 19912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 474.131666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 478.939220] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 479.217687] LustreError: 19913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 484.323648] LustreError: 19912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 487.234021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 492.113695] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 499.440423] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 504.791864] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 506.936685] Lustre: Failing over lustre-MDT0000 [ 507.165665] Lustre: server umount lustre-MDT0000 complete [ 507.876126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 507.883804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 507.896747] LustreError: 19913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.535139] Lustre: 3643:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787394229/real 1787394229] req@ffff95e784741180 x1874218256057600/t0(0) o400->MGC192.168.201.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787394245 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 525.550560] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 525.565913] LustreError: 19909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.577581] LustreError: 19909:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 526.971894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 535.072251] LustreError: 3641:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95e7badd9500 x1874218256060416/t0(0) o250->MGC192.168.201.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 539.693397] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 540.644374] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 540.665906] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 540.673634] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 548.846814] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 556.002671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.004445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 556.008220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 556.008234] Lustre: Skipped 1 previous similar message [ 556.017886] Lustre: Skipped 2 previous similar messages [ 559.792750] Lustre: server umount lustre-MDT0000 complete [ 562.144983] LustreError: 19907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.155674] LustreError: 19907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 564.034890] LustreError: 19893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394284 with bad export cookie 1694079264662163770 [ 564.043138] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 564.046622] LustreError: 19893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 564.328835] Lustre: server umount lustre-MDT0001 complete [ 573.413757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 574.125780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 574.136406] Lustre: Skipped 2 previous similar messages [ 578.734964] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 588.856485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 594.220307] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 602.317977] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 609.276472] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 612.761118] Lustre: Failing over lustre-MDT0000 [ 613.190756] Lustre: server umount lustre-MDT0000 complete [ 614.820973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 614.843174] Lustre: Skipped 1 previous similar message [ 630.239398] Lustre: 3642:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787394334/real 1787394334] req@ffff95e6879ec700 x1874218256087168/t0(0) o400->MGC192.168.201.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787394350 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 630.241459] LustreError: 24479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 630.269221] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 630.292066] LustreError: 24479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 634.101489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 641.962985] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 647.138741] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 647.150089] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 647.168340] Lustre: Skipped 1 previous similar message [ 647.188578] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 647.300351] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 658.492279] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 664.546526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 664.571020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 664.574020] Lustre: Skipped 4 previous similar messages [ 668.998764] Lustre: server umount lustre-MDT0000 complete [ 672.446037] LustreError: 23230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394392 with bad export cookie 1694079264662164708 [ 672.451952] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 672.459024] LustreError: 23230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 672.835234] Lustre: server umount lustre-MDT0001 complete [ 682.428287] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 06:26:41 (1787394401) [ 701.710091] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 714.471068] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 758.445104] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 769.132995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 769.400772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 769.429235] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 769.540459] Lustre: lustre-MDT0000: new disk, initializing [ 769.654077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 769.659284] Lustre: Skipped 2 previous similar messages [ 769.708441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 774.864721] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 786.027238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 786.142169] Lustre: 29626:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 786.370155] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 790.776479] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 800.199224] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 806.610573] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 816.804286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 817.365608] Lustre: 31592:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 818.682338] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 818.778452] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 818.938955] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 818.976077] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 825.031261] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 834.349372] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 840.121101] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 859.620596] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 859.629227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 859.639818] Lustre: Skipped 1 previous similar message [ 859.644594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 864.691501] Lustre: server umount lustre-OST0000 complete [ 874.943438] Lustre: server umount lustre-MDT0000 complete [ 875.498632] LustreError: 29631:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 875.524331] LustreError: 29631:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 879.517834] LustreError: 29617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394599 with bad export cookie 1694079264662165114 [ 879.526104] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 879.536736] LustreError: 29617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 880.537818] Lustre: server umount lustre-MDT0001 complete [ 894.073511] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 897.357348] Key type lgssc unregistered [ 897.734652] LNet: 33567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.752947] LNetError: 33567:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 897.764884] LNet: Removed LNI 192.168.201.114@tcp [ 898.760348] Key type .llcrypt unregistered [ 898.763641] Key type ._llcrypt unregistered [ 928.611817] Key type ._llcrypt registered [ 928.614546] Key type .llcrypt registered [ 928.752847] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 930.237676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 930.277788] alg: No test for adler32 (adler32-zlib) [ 931.326082] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 931.685834] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 933.375208] Key type lgssc registered [ 934.780313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 943.741182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 943.761604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 945.311561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 950.818711] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 955.873690] LustreError: 34981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 955.890745] LustreError: 34981:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 960.396232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 960.768057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 965.278519] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 972.871261] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 978.486040] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 985.349853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 985.617312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 990.713143] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 991.776382] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 996.852117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 999.509909] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1014.656888] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 06:32:13 (1787394733) [ 1016.217734] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1018.016306] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1025.042551] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 06:32:24 (1787394744) [ 1034.431811] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 06:32:33 (1787394753) [ 1035.539539] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1044.109438] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1044.123529] Lustre: Skipped 2 previous similar messages [ 1046.061808] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 06:32:45 (1787394765) [ 1049.546820] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1049.550996] Lustre: Skipped 3 previous similar messages [ 1066.061431] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1066.065515] Lustre: Skipped 6 previous similar messages [ 1068.545890] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 06:33:07 (1787394787) [ 1071.302884] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1074.595888] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1081.713636] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 06:33:21 (1787394801) [ 1083.384705] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1083.387920] Lustre: Skipped 3 previous similar messages [ 1096.126404] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 06:33:35 (1787394815) [ 1113.580123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1113.593049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1113.624450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1114.595179] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1114.598895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1114.612136] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1116.609664] Lustre: server umount lustre-OST0000 complete [ 1124.855201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1124.860849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1124.866587] Lustre: Skipped 1 previous similar message [ 1124.874954] Lustre: Skipped 1 previous similar message [ 1124.878192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1129.954579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1129.962046] Lustre: Skipped 2 previous similar messages [ 1135.071220] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1135.074064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1135.088762] Lustre: Skipped 1 previous similar message [ 1135.339263] Lustre: server umount lustre-MDT0000 complete [ 1138.826274] LustreError: 34961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394858 with bad export cookie 7328320098332611132 [ 1138.830745] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1138.832891] LustreError: 34961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1139.170336] Lustre: server umount lustre-MDT0001 complete [ 1149.670748] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 1152.532738] Key type lgssc unregistered [ 1152.955820] LNet: 42840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1152.962684] LNetError: 42840:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1152.976410] LNet: Removed LNI 192.168.201.114@tcp [ 1153.985089] Key type .llcrypt unregistered [ 1153.986920] Key type ._llcrypt unregistered [ 1171.723919] Key type ._llcrypt registered [ 1171.729784] Key type .llcrypt registered [ 1171.919958] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 1189.582800] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 1190.566269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1190.634059] alg: No test for adler32 (adler32-zlib) [ 1192.363746] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 1192.732762] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 1194.405350] Key type lgssc registered [ 1196.009081] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1234.458399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1234.489719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1235.918352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1235.944686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1236.040406] Lustre: lustre-MDT0000: new disk, initializing [ 1236.145899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1236.163539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1241.195799] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1253.086886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1253.301309] Lustre: 45997:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1253.361877] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1253.370362] Lustre: Skipped 1 previous similar message [ 1253.453198] Lustre: lustre-MDT0001: new disk, initializing [ 1253.561886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1253.594341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1253.617717] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1259.020945] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1268.585790] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1274.189304] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1281.304403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1281.792924] Lustre: lustre-OST0000: new disk, initializing [ 1281.797303] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1281.810858] Lustre: 47963:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1281.956035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1288.428465] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1290.882068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1290.901702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1291.012166] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1299.189869] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1305.014529] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1305.198615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1310.219603] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1310.436751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1316.835786] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1316.839752] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1316.866436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1318.916314] Lustre: server umount lustre-OST0000 complete [ 1325.546206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1325.559929] LustreError: Skipped 1 previous similar message [ 1325.566060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1325.572552] Lustre: Skipped 1 previous similar message [ 1325.576741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1325.580736] Lustre: Skipped 1 previous similar message [ 1327.074847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1327.077323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1329.071169] Lustre: server umount lustre-MDT0000 complete [ 1333.324284] LustreError: 46790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395053 with bad export cookie 5452073405101587698 [ 1333.326682] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1333.335523] LustreError: 46790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1334.132088] Lustre: server umount lustre-MDT0001 complete [ 1346.563727] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 06:37:45 (1787395065) [ 1353.969460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1354.506212] LustreError: 50426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1354.536408] LustreError: 50426:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1354.599676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1359.827779] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1359.844790] LustreError: 50427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1363.940565] LustreError: 50426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1368.906797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1369.062293] LustreError: 50427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1369.480294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1374.635273] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1382.733872] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1388.300992] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1395.457444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1395.889039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1402.558507] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1410.151424] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1418.410188] LustreError: 52955:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1419.487505] Lustre: Modifying parameter general.timeout=1 in log params [ 1420.542926] Lustre: Modifying parameter general.timeout=2 in log params [ 1423.298404] Lustre: Modifying parameter general.timeout=5 in log params [ 1423.302982] Lustre: Skipped 2 previous similar messages [ 1428.194401] Lustre: Modifying parameter general.timeout=10 in log params [ 1428.200047] Lustre: Skipped 4 previous similar messages [ 1436.455858] Lustre: Modifying parameter general.timeout=20 in log params [ 1436.460230] Lustre: Skipped 9 previous similar messages [ 1452.785393] Lustre: Modifying parameter general.timeout=34 in log params [ 1452.801438] Lustre: Skipped 13 previous similar messages [ 1485.662983] Lustre: Modifying parameter general.timeout=66 in log params [ 1485.669072] Lustre: Skipped 31 previous similar messages [ 1550.742029] Lustre: Disabling parameter general.timeout= in log params [ 1550.746858] Lustre: Skipped 63 previous similar messages [ 1554.101633] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 06:41:13 (1787395273) [ 1557.002508] Lustre: 59900:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1558.825813] Lustre: 60008:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1562.536531] Lustre: 60205:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1562.548202] Lustre: 60205:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1593.750667] Lustre: 61780:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1593.760451] Lustre: 61780:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1593.813392] LustreError: 61780:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1598.786606] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 06:41:57 (1787395317) [ 1615.329077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1615.338730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1615.348752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1615.365911] Lustre: Skipped 1 previous similar message [ 1620.289320] Lustre: server umount lustre-MDT0000 complete [ 1623.016666] LustreError: 50426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1623.054683] LustreError: 50426:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1624.548726] LustreError: 53401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395344 with bad export cookie 5452073405101588580 [ 1624.554912] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1624.564886] LustreError: 53401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1624.975545] Lustre: server umount lustre-MDT0001 complete [ 1639.634245] Lustre: server umount lustre-OST0000 complete [ 1670.281940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1670.486938] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1670.737935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1675.483268] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1683.264643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1683.322416] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1683.340623] Lustre: 64313:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1688.716295] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1695.898023] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1701.525960] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1708.811738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1709.052963] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1709.278966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1709.284206] Lustre: Skipped 1 previous similar message [ 1715.227381] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1721.747851] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1723.265986] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1723.270567] Lustre: Skipped 23 previous similar messages [ 1723.296909] Lustre: 66877:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1729.507915] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1729.510900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1729.534173] Lustre: Skipped 2 previous similar messages [ 1729.537145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1729.567102] Lustre: Skipped 4 previous similar messages [ 1735.506687] Lustre: server umount lustre-OST0000 complete [ 1739.747409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1739.757293] LustreError: Skipped 1 previous similar message [ 1739.766453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1739.773582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1739.785285] Lustre: Skipped 2 previous similar messages [ 1739.796163] Lustre: Skipped 3 previous similar messages [ 1745.703431] Lustre: server umount lustre-MDT0000 complete [ 1749.737920] LustreError: 64304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395469 with bad export cookie 5452073405101591702 [ 1749.756881] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1749.760733] LustreError: 64304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1750.193099] Lustre: server umount lustre-MDT0001 complete [ 1761.472874] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 1764.647680] Key type lgssc unregistered [ 1764.954553] LNet: 68088:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1764.957668] LNetError: 68088:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.984982] LNet: Removed LNI 192.168.201.114@tcp [ 1765.920276] Key type .llcrypt unregistered [ 1765.923593] Key type ._llcrypt unregistered [ 1792.886679] Key type ._llcrypt registered [ 1792.889565] Key type .llcrypt registered [ 1793.056270] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 1794.056356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1794.244047] alg: No test for adler32 (adler32-zlib) [ 1795.459613] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 1795.781718] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 1797.543199] Key type lgssc registered [ 1798.626165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1807.580467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1807.604122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1809.058392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1815.192233] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1819.628777] LustreError: 69519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1819.636956] LustreError: 69519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1824.494443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1824.904714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1830.156413] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1837.361290] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1843.825270] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1850.695449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1851.344976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1857.707361] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1865.398495] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1876.375212] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1886.177106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1886.184388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1886.196688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1890.274267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1890.279449] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1890.282260] Lustre: Skipped 1 previous similar message [ 1890.305909] Lustre: Skipped 1 previous similar message [ 1891.856313] Lustre: server umount lustre-MDT0000 complete [ 1895.261565] LustreError: 69499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395615 with bad export cookie 13961281277777779373 [ 1895.263821] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1895.266834] LustreError: 69499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1895.395629] LustreError: 69514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1895.408516] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1895.522773] Lustre: server umount lustre-MDT0001 complete [ 1901.545304] Lustre: server umount lustre-OST0000 complete [ 1927.092106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1927.368996] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1927.395673] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1927.668137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1932.018901] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1940.808173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1940.928186] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1940.980134] Lustre: 74390:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1941.241557] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1941.258918] Lustre: Skipped 1 previous similar message [ 1941.532360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1947.189208] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1955.805215] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1963.477478] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1972.722344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1972.865521] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1973.405396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1980.509480] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1986.193657] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1987.873050] Lustre: 76970:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1989.371818] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1994.212394] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1994.223511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1994.247828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1999.862116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1999.865959] Lustre: Skipped 2 previous similar messages [ 2000.391332] Lustre: server umount lustre-OST0000 complete [ 2004.960707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2004.988035] Lustre: Skipped 2 previous similar messages [ 2009.897988] Lustre: server umount lustre-MDT0000 complete [ 2010.086990] LustreError: 75855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2010.106144] LustreError: 75855:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2013.336405] LustreError: 74383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395733 with bad export cookie 13961281277777780213 [ 2013.344239] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2013.349345] LustreError: 74383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2013.680770] Lustre: server umount lustre-MDT0001 complete [ 2023.587217] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 2026.044655] Key type lgssc unregistered [ 2026.330843] LNet: 78179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2026.337473] LNetError: 78179:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2026.357156] LNet: Removed LNI 192.168.201.114@tcp [ 2027.267670] Key type .llcrypt unregistered [ 2027.269423] Key type ._llcrypt unregistered [ 2049.111288] Key type ._llcrypt registered [ 2049.116469] Key type .llcrypt registered [ 2049.197241] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2050.051708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2050.130172] alg: No test for adler32 (adler32-zlib) [ 2051.205410] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 2051.435936] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 2053.080884] Key type lgssc registered [ 2054.204819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2060.941958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2060.960915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2062.444805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2066.339629] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2072.034136] LustreError: 79591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2072.049530] LustreError: 79591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2073.376524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.897687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2077.503121] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2083.502939] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2087.965960] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2094.228484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2094.646207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2099.694328] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2105.298813] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2115.780864] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2116.558031] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2118.198094] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2118.205958] Lustre: Skipped 1 previous similar message [ 2120.733949] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2120.738092] Lustre: Skipped 2 previous similar messages [ 2124.858715] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2124.865293] Lustre: Skipped 4 previous similar messages [ 2133.482605] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2133.485459] Lustre: Skipped 11 previous similar messages [ 2149.597827] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2149.601462] Lustre: Skipped 22 previous similar messages [ 2182.018552] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2182.021022] Lustre: Skipped 37 previous similar messages [ 2199.477326] Lustre: 87688:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2202.168686] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 06:52:01 (1787395921) [ 2204.719590] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2206.833189] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 06:52:05 (1787395925) [ 2214.381262] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2214.384329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2214.395928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2214.410893] Lustre: Skipped 1 previous similar message [ 2217.739685] Lustre: server umount lustre-OST0000 complete [ 2224.616532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2224.632408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2224.640207] Lustre: Skipped 2 previous similar messages [ 2229.352616] Lustre: server umount lustre-MDT0000 complete [ 2229.740693] LustreError: 79588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2229.765489] LustreError: 79588:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2233.692642] LustreError: 84013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395953 with bad export cookie 12770906365325261647 [ 2233.694861] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2233.699208] LustreError: 84013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2234.164564] Lustre: server umount lustre-MDT0001 complete [ 2246.974091] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 2249.969493] Key type lgssc unregistered [ 2250.377745] LNet: 88937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2250.403715] LNetError: 88937:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2250.426890] LNet: Removed LNI 192.168.201.114@tcp [ 2251.182045] Key type .llcrypt unregistered [ 2251.188715] Key type ._llcrypt unregistered [ 2255.496656] Key type ._llcrypt registered [ 2255.499704] Key type .llcrypt registered [ 2255.585277] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2260.480212] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules [ 2261.791724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2261.850503] alg: No test for adler32 (adler32-zlib) [ 2262.976682] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 2263.168594] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 2264.903187] Key type lgssc registered [ 2266.089316] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2267.347713] LustreError: 90140:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2287.096432] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2289.754625] LustreError: 90140:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2296.778733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2296.802995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2298.551575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2303.734870] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2308.069684] LustreError: 91162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2308.109153] LustreError: 91162:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2311.837575] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 06:53:51 (1787396031) [ 2313.193510] LustreError: 91163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2316.103828] LustreError: 91194:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2316.120614] LustreError: 91194:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2316.132629] LustreError: 91194:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2316.447884] Lustre: server umount lustre-MDT0000 complete [ 2329.907903] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 2332.369608] Key type lgssc unregistered [ 2332.625161] LNet: 92687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2332.629768] LNetError: 92687:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2333.670133] LNet: Removed LNI 192.168.201.114@tcp [ 2334.427669] Key type .llcrypt unregistered [ 2334.432137] Key type ._llcrypt unregistered [ 2350.956407] Key type ._llcrypt registered [ 2350.959149] Key type .llcrypt registered [ 2351.039491] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2352.051220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2352.095063] alg: No test for adler32 (adler32-zlib) [ 2353.119574] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 2353.277650] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 2354.944907] Key type lgssc registered [ 2356.001304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2363.158978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2363.171727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2364.721327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2369.088720] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2374.121061] LustreError: 93907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.146096] LustreError: 93907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2377.310356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2377.922291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2382.289419] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2389.235759] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2394.668145] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2400.920941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2401.521409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2408.102329] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2415.115984] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2459.844270] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 06:56:18 (1787396178) [ 2461.700406] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2463.656966] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 06:56:22 (1787396182) [ 2469.867716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2469.881522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2469.896728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2471.394373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.408814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2471.415184] Lustre: Skipped 1 previous similar message [ 2471.426515] Lustre: Skipped 2 previous similar messages [ 2474.596471] Lustre: server umount lustre-MDT0000 complete [ 2476.515264] LustreError: 93907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2476.544721] LustreError: 93907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2478.263862] LustreError: 93888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396198 with bad export cookie 14729485227520465225 [ 2478.265217] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2478.275756] LustreError: 93888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2478.629642] Lustre: server umount lustre-MDT0001 complete [ 2493.166299] Lustre: server umount lustre-OST0000 complete [ 2504.225822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2504.719190] LustreError: 98184:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2504.910309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2509.052364] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2510.305054] LustreError: 98185:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2514.411072] LustreError: 98184:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2517.382616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2518.026553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2522.784386] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2530.275191] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2536.080979] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2549.155551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2549.385867] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2549.397157] LustreError: 98173:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2549.443587] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2549.456538] LustreError: 100268:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2549.467966] LustreError: 100268:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2549.476718] LustreError: 100268:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2549.575215] Lustre: server umount lustre-OST0000 complete [ 2549.577693] LustreError: 100268:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2557.721927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2557.897747] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2557.905616] LustreError: 98172:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2557.926815] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2557.936209] LustreError: 100734:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2557.941931] LustreError: 100734:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2557.947327] LustreError: 100734:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2558.010446] Lustre: server umount lustre-OST0000 complete [ 2558.012475] LustreError: 100734:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2566.807279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2567.027231] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2567.032411] Lustre: Found index 0 for lustre-OST0000, updating log [ 2567.043931] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2567.188480] Lustre: lustre-OST0000: new disk, initializing [ 2567.196869] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2567.205816] Lustre: 101268:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2567.498606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2573.708767] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2577.705305] Lustre: Failing over lustre-OST0000 [ 2577.964629] Lustre: server umount lustre-OST0000 complete [ 2584.546445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2584.552989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2584.570151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2587.111174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2587.115356] Lustre: Skipped 3 previous similar messages [ 2592.225110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2592.232980] Lustre: Skipped 1 previous similar message [ 2596.319131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2596.480294] Lustre: server umount lustre-MDT0000 complete [ 2597.349201] LustreError: 98181:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2597.371615] LustreError: 98181:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2599.722708] LustreError: 98166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396319 with bad export cookie 14729485227520467969 [ 2599.727048] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2599.736577] LustreError: 98166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2600.094411] Lustre: server umount lustre-MDT0001 complete [ 2608.370981] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 06:58:47 (1787396327) [ 2625.266930] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2637.579934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2638.180474] LustreError: 103612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2638.405452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2645.754684] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2656.964667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2663.446189] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2666.448632] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2672.973805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2673.820237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2673.829212] Lustre: Skipped 1 previous similar message [ 2673.855604] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2680.173541] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2682.020966] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2682.024910] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2682.032168] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2683.048671] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:09 [ 2683.085765] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2683.088861] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2683.123456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2688.618695] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2688.722298] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2688.821933] Lustre: lustre-OST0001: new disk, initializing [ 2688.826897] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2688.835428] Lustre: 105934:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2694.755235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2694.766743] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2694.924271] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2695.971651] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2703.487348] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2709.901891] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2714.600579] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2714.618651] LustreError: Skipped 1 previous similar message [ 2714.625630] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2714.653599] Lustre: Skipped 1 previous similar message [ 2714.660801] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2720.525990] Lustre: server umount lustre-MDT0001 complete [ 2720.737190] LustreError: 104344:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2720.748166] LustreError: 104344:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2732.099069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2732.201139] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2732.210949] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2732.269356] Lustre: 103601:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2733.958418] LustreError: 107467:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2733.971602] LustreError: 107467:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2733.981066] LustreError: 103605:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2734.095754] LustreError: 107473:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.114@tcp: cfg command failed: rc = -17 [ 2734.105861] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2734.105861] [ 2738.705035] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2739.000992] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2751.734995] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 07:01:10 (1787396470) [ 2770.601980] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2793.495726] Lustre: 109560:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2815.540620] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2819.370162] Lustre: 110696:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2901.475202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2901.486804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2901.514264] Lustre: Skipped 2 previous similar messages [ 2901.518865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2901.526548] Lustre: Skipped 4 previous similar messages [ 2905.661403] Lustre: server umount lustre-MDT0000 complete [ 2907.616776] LustreError: 103609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2907.654111] LustreError: 103609:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 2910.087794] LustreError: 105132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396630 with bad export cookie 14729485227520468970 [ 2910.090155] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2910.095081] LustreError: 105132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2910.352892] Lustre: server umount lustre-MDT0001 complete [ 2925.276446] Lustre: server umount lustre-OST0000 complete [ 2928.607103] Lustre: 93220:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787396632/real 1787396632] req@ffff95e68489c000 x1874220625007616/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787396648 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2928.654238] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2928.674127] Lustre: Skipped 3 previous similar messages [ 2929.320535] Lustre: server umount lustre-OST0001 complete [ 2934.676412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2945.247267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2955.358586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2966.201532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2989.387349] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 3000.140811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3000.155413] Lustre: lustre-MDT0000: reset Object Index mappings [ 3000.596522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3000.599875] Lustre: Skipped 2 previous similar messages [ 3005.130470] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3014.572910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3014.590193] Lustre: lustre-MDT0001: reset Object Index mappings [ 3020.946370] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3024.108708] Lustre: 115807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3031.126212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3037.924958] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3041.150098] LustreError: 116161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3041.177523] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3041.185019] LustreError: 116161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3046.280783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3048.158337] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3054.084947] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3054.097965] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3056.239426] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3062.709342] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3065.633425] Lustre: 117685:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3109.612501] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 07:07:09 (1787396829) [ 3138.017371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3138.032144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3138.060490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3138.072723] Lustre: Skipped 4 previous similar messages [ 3142.779552] Lustre: server umount lustre-MDT0000 complete [ 3146.450737] LustreError: 114640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396866 with bad export cookie 14729485227520516087 [ 3146.466745] LustreError: 114640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3146.473501] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3152.870437] Lustre: server umount lustre-MDT0001 complete [ 3163.436891] Lustre: server umount lustre-OST0000 complete [ 3173.949278] Lustre: server umount lustre-OST0001 complete [ 3181.817239] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 3194.664376] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 3232.423251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3232.775495] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3232.816315] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3232.915614] Lustre: lustre-MDT0000: new disk, initializing [ 3233.113774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3233.125220] Lustre: Skipped 3 previous similar messages [ 3233.169624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3238.390316] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3246.265475] Lustre: server umount lustre-MDT0000 complete [ 3258.049759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3258.528042] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3258.564166] Lustre: 121954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3258.595534] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3264.253146] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3269.886975] Lustre: server umount lustre-MDT0000 complete [ 3279.633756] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 07:09:58 (1787396998) [ 3281.902847] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3283.936811] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 07:10:03 (1787397003) [ 3302.568754] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 3312.176672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3318.509812] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3330.873783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3330.956918] Lustre: 124008:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3331.020823] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3331.024185] Lustre: Skipped 1 previous similar message [ 3331.093246] Lustre: lustre-MDT0001: new disk, initializing [ 3331.167596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3331.181610] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3336.562753] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3341.162435] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3348.433877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3348.770471] Lustre: lustre-OST0000: new disk, initializing [ 3348.781757] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3348.794494] Lustre: 125587:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3350.786112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3350.798378] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3350.921907] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3355.641966] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3367.169834] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3367.462604] Lustre: lustre-OST0001: new disk, initializing [ 3367.466828] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3367.486455] Lustre: 126456:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3369.522658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3369.530495] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3369.698383] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3374.769384] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3386.048257] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3390.254116] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3775.084474] Lustre: 132867:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3775.093255] Lustre: 132867:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3775.098882] Lustre: 132867:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3775.104988] Lustre: 132867:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3775.112431] Lustre: 132867:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3775.117108] Lustre: 132867:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4027.950542] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:22:26 (1787397746) [ 4045.806710] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4045.821473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4045.828420] Lustre: Skipped 8 previous similar messages [ 4045.844345] Lustre: Skipped 9 previous similar messages [ 4049.814281] Lustre: server umount lustre-MDT0000 complete [ 4050.927929] LustreError: 124832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4050.942456] LustreError: 124832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4053.738103] LustreError: 124755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397773 with bad export cookie 14729485227520581166 [ 4053.749162] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4053.750965] LustreError: 124755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4056.034732] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4056.042224] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4056.079904] Lustre: Skipped 1 previous similar message [ 4068.320569] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4068.710520] Lustre: server umount lustre-MDT0001 complete [ 4078.453063] Lustre: server umount lustre-OST0000 complete [ 4087.621183] Lustre: server umount lustre-OST0001 complete [ 4095.159051] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 4106.897554] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4145.035478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4145.329430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4145.379942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4145.486729] Lustre: lustre-MDT0000: new disk, initializing [ 4145.661729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4145.671120] Lustre: Skipped 5 previous similar messages [ 4145.698633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4150.857513] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4162.257457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4162.501936] Lustre: 135911:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4162.632553] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4162.640021] Lustre: Skipped 1 previous similar message [ 4162.763252] Lustre: lustre-MDT0001: new disk, initializing [ 4163.006330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4163.021339] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4169.107465] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4180.079432] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4186.296678] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4193.459453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4193.938854] Lustre: lustre-OST0000: new disk, initializing [ 4193.947359] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4193.968037] Lustre: 137880:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4194.151945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4194.159622] Lustre: Skipped 1 previous similar message [ 4195.271649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4195.283248] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4195.486824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4200.429362] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4209.848112] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4220.050882] Lustre: lustre-MDD0000: changelog on [ 4222.970964] Lustre: lustre-MDD0001: changelog on [ 5927.024879] Lustre: lustre-MDD0001: changelog off [ 5929.260177] Lustre: 135919:0:(llog_cat.c:984:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5930.200954] Lustre: lustre-MDD0000: changelog off [ 5936.610058] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5936.631625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5936.653136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5936.657687] Lustre: Skipped 7 previous similar messages [ 5941.220851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5941.223302] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5941.224938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5942.540366] Lustre: server umount lustre-OST0000 complete [ 5949.922502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5949.932946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5949.980941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5949.996086] Lustre: Skipped 1 previous similar message [ 5953.913545] Lustre: server umount lustre-MDT0000 complete [ 5956.578959] LustreError: 135920:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5956.596493] LustreError: 135920:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 5958.168077] LustreError: 135904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399678 with bad export cookie 14729485227521360140 [ 5958.169291] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5958.176304] LustreError: 135904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5958.496645] Lustre: server umount lustre-MDT0001 complete [ 5970.364413] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 5973.349640] Key type lgssc unregistered [ 5973.714996] LNet: 147000:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5973.720923] LNetError: 147000:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5973.733819] LNet: Removed LNI 192.168.201.114@tcp [ 5974.979250] Key type .llcrypt unregistered [ 5974.983671] Key type ._llcrypt unregistered [ 5996.455882] Key type ._llcrypt registered [ 5996.458853] Key type .llcrypt registered [ 5996.623450] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 6012.732977] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 6013.817366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6013.866453] alg: No test for adler32 (adler32-zlib) [ 6015.114340] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 6015.370452] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 6017.071928] Key type lgssc registered [ 6018.789785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6061.341171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6061.386582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6062.937318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6062.991976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6063.130289] Lustre: lustre-MDT0000: new disk, initializing [ 6063.255834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6063.307428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6068.547290] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6080.084616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6080.260870] Lustre: 150204:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6080.322714] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6080.326761] Lustre: Skipped 1 previous similar message [ 6080.433200] Lustre: lustre-MDT0001: new disk, initializing [ 6080.530695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6080.560502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6080.580680] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6085.420643] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6094.013496] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6099.429848] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6106.177550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6106.414251] Lustre: lustre-OST0000: new disk, initializing [ 6106.418795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6106.424352] Lustre: 152174:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6106.524833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6113.811632] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6116.424166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6116.429953] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6116.545164] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6123.248281] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6129.263432] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_module obdecho/obdecho [ 6130.879481] LustreError: 153178:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6130.887346] LustreError: 153178:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6132.596559] LustreError: 153274:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 6147.055797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6147.059905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6147.083948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6151.055583] Lustre: server umount lustre-MDT0000 complete [ 6156.320973] LustreError: 150198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399876 with bad export cookie 14518904410674263973 [ 6156.342123] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6157.285336] Lustre: server umount lustre-MDT0001 complete [ 6171.152474] Lustre: server umount lustre-OST0000 complete [ 6196.519436] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 6209.607590] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 6248.819368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6249.064701] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6249.087549] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6249.186684] Lustre: lustre-MDT0000: new disk, initializing [ 6249.270603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6249.285822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6254.285620] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6264.690535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6264.821327] Lustre: 156990:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6264.892226] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6264.896608] Lustre: Skipped 1 previous similar message [ 6265.013187] Lustre: lustre-MDT0001: new disk, initializing [ 6265.134272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6265.191760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6265.201372] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6270.655431] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6281.128745] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6286.865619] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6293.708579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6294.045996] Lustre: lustre-OST0000: new disk, initializing [ 6294.051908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6294.061642] Lustre: 158961:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6294.145555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6300.322679] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6303.824890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6303.838876] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6303.967900] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6309.605675] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6314.734615] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6314.984843] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6320.572611] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6320.944944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6324.705470] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6324.708150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6324.725560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6324.725568] Lustre: Skipped 2 previous similar messages [ 6324.768564] Lustre: Skipped 3 previous similar messages [ 6329.408643] Lustre: server umount lustre-OST0000 complete [ 6334.946015] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6334.951419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6334.972287] Lustre: Skipped 1 previous similar message [ 6334.985481] Lustre: Skipped 2 previous similar messages [ 6340.066440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6340.073767] Lustre: Skipped 1 previous similar message [ 6344.162835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6344.171719] Lustre: Skipped 1 previous similar message [ 6348.256954] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6348.439572] Lustre: server umount lustre-MDT0000 complete [ 6349.284871] LustreError: 157002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6349.309390] LustreError: 157002:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6352.748954] LustreError: 156982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400072 with bad export cookie 14518904410674264932 [ 6352.764571] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6352.769936] LustreError: 156982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6353.175481] Lustre: server umount lustre-MDT0001 complete [ 6357.314600] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 08:01:16 (1787400076) [ 6359.101874] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6361.433932] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 08:01:20 (1787400080) [ 6370.682653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6371.341432] LustreError: 161384:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6371.448501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6376.933109] LustreError: 161385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6377.048933] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6381.091664] LustreError: 161384:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6386.153037] LustreError: 161385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6387.575759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6393.005218] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6401.219118] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6407.011933] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6414.719628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6415.201099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6415.219910] Lustre: Skipped 1 previous similar message [ 6422.552457] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6431.015373] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6444.005991] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6444.008122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6444.029805] Lustre: Skipped 1 previous similar message [ 6448.819435] Lustre: server umount lustre-MDT0000 complete [ 6449.121606] LustreError: 161379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6449.130029] LustreError: 161379:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6452.885710] LustreError: 161366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400172 with bad export cookie 14518904410674265814 [ 6452.899538] LustreError: 161366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6452.909887] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6454.248300] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6454.265556] Lustre: Skipped 1 previous similar message [ 6459.328162] Lustre: server umount lustre-MDT0001 complete [ 6465.931426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6478.473854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6478.667147] LustreError: 165226:0:(llog.c:1655:llog_backup()) MGC192.168.201.114@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6478.674265] Lustre: 165226:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.114@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6482.848476] LustreError: 165231:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6482.861130] LustreError: 165231:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6518.753581] LustreError: 165232:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6518.778050] LustreError: 165232:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 28 previous similar messages [ 6518.878364] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.14@tcp (not set up) [ 6518.887081] Lustre: Skipped 6 previous similar messages [ 6539.231675] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6539.234972] Lustre: 165226:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.114@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6580.218766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6584.419103] LustreError: 165232:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6584.441989] LustreError: 165232:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 41 previous similar messages [ 6585.139198] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6585.322722] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6594.783255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6598.648578] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc97d82a31617980c [ 6598.657444] Lustre: MGC192.168.201.114@tcp: Connection restored to 0@lo (at 0@lo) [ 6598.755524] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.14@tcp (not set up) [ 6604.270954] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6604.572343] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6612.380945] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6614.009750] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6619.335715] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6621.258140] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6640.137297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6640.147392] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6640.178183] Lustre: Skipped 2 previous similar messages [ 6645.009876] Lustre: server umount lustre-MDT0000 complete [ 6648.914307] LustreError: 161365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400369 with bad export cookie 14518904410674272268 [ 6648.923582] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6648.926565] LustreError: 161365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6649.481657] Lustre: server umount lustre-MDT0001 complete [ 6663.683664] Lustre: server umount lustre-OST0000 complete [ 6687.295782] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 6695.470854] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 6735.776279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6736.060272] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6736.082995] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6736.170202] Lustre: lustre-MDT0000: new disk, initializing [ 6736.264551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6736.272405] Lustre: Skipped 1 previous similar message [ 6736.286378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6740.961190] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6753.291332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6753.483706] Lustre: 170333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6753.946323] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6761.237618] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6771.900377] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6777.577198] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6785.274025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6785.544391] Lustre: lustre-OST0000: new disk, initializing [ 6785.547584] Lustre: Skipped 1 previous similar message [ 6785.550819] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6785.557341] Lustre: Skipped 2 previous similar messages [ 6785.563737] Lustre: 172305:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6790.763847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6790.773723] Lustre: Skipped 1 previous similar message [ 6790.779336] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6790.889557] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6792.189218] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6802.694893] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6808.493506] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6808.741542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6814.429178] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6814.762818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6821.867572] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6821.878829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6821.879751] LustreError: Skipped 1 previous similar message [ 6821.883168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6821.883176] Lustre: Skipped 3 previous similar messages [ 6821.923557] Lustre: Skipped 1 previous similar message [ 6823.522848] Lustre: server umount lustre-OST0000 complete [ 6830.560546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6834.296291] Lustre: server umount lustre-MDT0000 complete [ 6837.230247] LustreError: 171197:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6837.255088] LustreError: 171197:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 6839.858738] LustreError: 170324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400559 with bad export cookie 14518904410674289103 [ 6839.861145] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6839.865343] LustreError: 170324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6840.255943] Lustre: server umount lustre-MDT0001 complete [ 6850.519883] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 08:09:29 (1787400569) [ 6857.193843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6862.365562] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6871.384551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6877.244104] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6884.893516] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6889.763817] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6895.876417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6901.823692] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6909.759674] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6919.899104] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 08:10:39 (1787400639) [ 6927.840176] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6927.840525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6927.850938] LustreError: Skipped 1 previous similar message [ 6927.870777] Lustre: Skipped 3 previous similar messages [ 6930.140493] Lustre: server umount lustre-OST0000 complete [ 6940.758635] Lustre: server umount lustre-MDT0000 complete [ 6944.540084] LustreError: 174697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400664 with bad export cookie 14518904410674289985 [ 6944.562060] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6944.567318] LustreError: 174697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6944.927557] Lustre: server umount lustre-MDT0001 complete [ 6957.864415] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 6961.034516] Key type lgssc unregistered [ 6961.326977] LNet: 178446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6961.336668] LNetError: 178446:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6962.406648] LNet: Removed LNI 192.168.201.114@tcp [ 6963.803789] Key type .llcrypt unregistered [ 6963.806915] Key type ._llcrypt unregistered [ 6985.380447] Key type ._llcrypt registered [ 6985.384150] Key type .llcrypt registered [ 6985.495289] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 6986.522344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6986.686799] alg: No test for adler32 (adler32-zlib) [ 6987.858152] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 6988.133243] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 6989.791192] Key type lgssc registered [ 6991.109884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6998.777498] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6998.800711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7014.367499] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7019.489784] LustreError: 179691:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.114@tcp: failed processing log, type 4: rc = -110 [ 7024.608570] LustreError: 179691:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7024.624823] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7024.632995] LustreError: MGC192.168.201.114@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7024.664546] LustreError: 179691:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7024.680500] LustreError: 179691:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7024.697858] LustreError: 179691:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7024.769318] Lustre: server umount lustre-OST0000 complete [ 7024.775187] LustreError: 179691:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7032.797337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7034.184945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7039.850670] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7044.578869] LustreError: 180191:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7044.609723] LustreError: 180191:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7049.714655] LustreError: 180192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7051.299963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7051.794813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7056.553222] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7065.565870] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7072.371213] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7079.734324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7079.958847] LustreError: 182009:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7079.968956] Lustre: 182009:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.114@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7080.151945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7087.767611] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7095.468745] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7097.826205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7097.830143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7097.859671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7101.932712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7101.941773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7101.944870] Lustre: Skipped 1 previous similar message [ 7101.954607] Lustre: Skipped 2 previous similar messages [ 7107.044905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7107.057777] Lustre: Skipped 2 previous similar messages [ 7112.160745] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7112.164264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7112.185079] Lustre: Skipped 2 previous similar messages [ 7112.378735] Lustre: server umount lustre-MDT0000 complete [ 7117.217131] LustreError: 180171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400837 with bad export cookie 8663459061654509999 [ 7117.221041] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7117.234626] LustreError: 180171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7117.285669] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7117.297607] LustreError: 182682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7117.315233] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7117.944293] Lustre: server umount lustre-MDT0001 complete [ 7125.003924] Lustre: server umount lustre-OST0000 complete [ 7136.597636] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 7155.464098] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 7204.640687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7204.952532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7205.029301] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7205.129697] Lustre: lustre-MDT0000: new disk, initializing [ 7205.294804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7205.354895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7211.936040] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7225.294278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7225.472820] Lustre: 185560:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7225.568039] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7225.577091] Lustre: Skipped 1 previous similar message [ 7225.696818] Lustre: lustre-MDT0001: new disk, initializing [ 7225.769388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7225.804455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7225.819919] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7231.085329] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7242.337616] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7249.299711] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7257.420795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7257.738397] Lustre: lustre-OST0000: new disk, initializing [ 7257.743581] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7257.755608] Lustre: 187529:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7257.890690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7265.270759] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7266.909141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7266.920958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7267.063423] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7274.731509] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7279.620392] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7279.883422] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7284.944379] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7285.186545] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7287.779441] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7287.781659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7287.792505] LustreError: Skipped 1 previous similar message [ 7287.794270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7287.812340] Lustre: Skipped 1 previous similar message [ 7293.475365] Lustre: server umount lustre-OST0000 complete [ 7302.624156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7302.635157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7308.258047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7308.263969] Lustre: Skipped 7 previous similar messages [ 7312.351372] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7312.645968] Lustre: server umount lustre-MDT0000 complete [ 7313.383706] LustreError: 185566:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7313.415929] LustreError: 185566:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7316.777147] LustreError: 185552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401036 with bad export cookie 8663459061654510713 [ 7316.788697] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7317.099822] Lustre: server umount lustre-MDT0001 complete [ 7326.456224] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 08:17:25 (1787401045) [ 7346.355214] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 7348.934839] Key type lgssc unregistered [ 7349.221564] LNet: 190478:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7349.228461] LNetError: 190478:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7349.240996] LNet: Removed LNI 192.168.201.114@tcp [ 7350.003044] Key type .llcrypt unregistered [ 7350.006320] Key type ._llcrypt unregistered [ 7377.458314] Key type ._llcrypt registered [ 7377.459823] Key type .llcrypt registered [ 7377.533471] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 7378.627523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7378.823036] alg: No test for adler32 (adler32-zlib) [ 7379.827979] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 7380.023968] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 7381.655184] Key type lgssc registered [ 7383.246960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7392.754888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7392.771625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7394.260446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7399.894146] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7404.515489] LustreError: 191696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7404.532290] LustreError: 191696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7408.985421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7409.480313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7414.824555] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7422.700540] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7429.138762] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7437.735656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7437.980463] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7437.982505] LustreError: 193560:0:(llog.c:1696:llog_backup()) MGC192.168.201.114@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7437.985759] Lustre: 193560:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.114@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7438.085015] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7445.488637] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7453.763524] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7459.819404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7459.837320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7459.847655] Lustre: Skipped 1 previous similar message [ 7459.865598] Lustre: Skipped 2 previous similar messages [ 7462.805662] Lustre: server umount lustre-MDT0000 complete [ 7464.933744] LustreError: 192675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7464.952933] LustreError: 192675:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7466.344932] LustreError: 191678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401186 with bad export cookie 5907083241704518928 [ 7466.354762] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7466.373650] LustreError: 191678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7466.856680] Lustre: server umount lustre-MDT0001 complete [ 7482.490980] Lustre: server umount lustre-OST0000 complete [ 7495.016358] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 7510.923275] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 7560.247206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7560.728267] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7560.788603] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7560.962393] Lustre: lustre-MDT0000: new disk, initializing [ 7561.106849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7561.129385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7568.432200] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7581.134497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7581.326863] Lustre: 197107:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7581.466268] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7581.476334] Lustre: Skipped 1 previous similar message [ 7581.479703] LustreError: 197893:0:(llog.c:1696:llog_backup()) MGC192.168.201.114@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7581.500506] LustreError: 197893:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7581.515354] Lustre: 197893:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.114@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7581.558203] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7581.565092] Lustre: Skipped 1 previous similar message [ 7581.693471] Lustre: lustre-MDT0001: new disk, initializing [ 7581.791171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7581.835104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7581.845410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7587.277946] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7598.563228] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7604.862662] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7612.788145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7613.105332] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7613.108466] Lustre: Skipped 1 previous similar message [ 7613.113920] LustreError: 199056:0:(llog.c:1696:llog_backup()) MGC192.168.201.114@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7613.122129] LustreError: 199056:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7613.133706] Lustre: 199056:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.114@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7613.139338] Lustre: 199056:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7613.160719] Lustre: lustre-OST0000: new disk, initializing [ 7613.165431] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7613.171370] Lustre: 199081:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7613.268244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7621.377206] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7621.744147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7621.764350] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7621.869308] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7631.906284] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7638.181087] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7638.509937] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7644.481103] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7644.719634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7647.750535] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7647.757838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7647.758709] LustreError: Skipped 1 previous similar message [ 7647.759416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7647.805058] Lustre: Skipped 2 previous similar messages [ 7652.849866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7652.867786] Lustre: Skipped 2 previous similar messages [ 7653.032691] Lustre: server umount lustre-OST0000 complete [ 7657.962371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7657.968221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7657.981773] Lustre: Skipped 1 previous similar message [ 7663.075939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7663.084282] Lustre: Skipped 2 previous similar messages [ 7663.450209] Lustre: server umount lustre-MDT0000 complete [ 7667.670345] LustreError: 199080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401387 with bad export cookie 5907083241704519656 [ 7667.680808] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7667.683217] LustreError: 199080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7668.267485] Lustre: server umount lustre-MDT0001 complete [ 7678.193599] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 08:23:17 (1787401397) [ 7697.508720] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 7709.281665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7709.831768] LustreError: 202209:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7709.904117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7715.212752] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7715.296352] LustreError: 202210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7719.422990] LustreError: 202209:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7724.202840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7730.473721] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7734.178214] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7742.580561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7743.073824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7743.083367] Lustre: Skipped 1 previous similar message [ 7749.770815] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7760.332934] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7760.656632] Lustre: lustre-OST0001: new disk, initializing [ 7760.668482] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7760.678439] Lustre: 204514:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7768.163068] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7768.195827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7768.205048] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7768.403156] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7779.338617] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7786.281353] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7789.225082] Lustre: DEBUG MARKER: ADD OST3 [ 7794.731399] loop: module loaded [ 7794.822914] loop0: detected capacity change from 0 to 204800000 [ 7795.186462] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7795.193926] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7795.202734] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7800.453303] loop0: detected capacity change from 0 to 204800000 [ 7801.416784] Lustre: DEBUG MARKER: STOP OST3 [ 7807.929729] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7808.199654] Lustre: lustre-OST0003: new disk, initializing [ 7808.226045] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7808.235110] Lustre: 206502:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7808.501716] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7808.517331] Lustre: Skipped 1 previous similar message [ 7810.298947] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7810.498908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7810.518682] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7810.577720] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7812.330783] LustreError: 206648:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7813.301891] Lustre: Failing over lustre-OST0003 [ 7813.464543] Lustre: server umount lustre-OST0003 complete [ 7814.806775] LustreError: 203693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7815.677709] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7815.695935] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7815.722642] Lustre: Skipped 1 previous similar message [ 7816.930837] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7827.385804] Lustre: DEBUG MARKER: START OST3 again [ 7831.024806] LustreError: 204881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7831.046877] LustreError: 204881:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 7835.201309] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7835.705909] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7837.061139] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7837.717834] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7837.721270] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7843.429514] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7864.155692] Lustre: Failing over lustre-OST0003 [ 7864.296963] Lustre: server umount lustre-OST0003 complete [ 7866.352243] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7866.386377] LustreError: 204881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7866.408526] LustreError: 204881:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7883.232369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7883.242423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7883.251482] Lustre: Skipped 1 previous similar message [ 7883.256149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7888.846888] Lustre: server umount lustre-MDT0000 complete [ 7892.307984] LustreError: 205283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401612 with bad export cookie 5907083241704520538 [ 7892.309831] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7892.317612] LustreError: 205283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7899.379703] Lustre: server umount lustre-MDT0001 complete [ 7904.434414] Lustre: server umount lustre-OST0000 complete [ 7909.133861] Lustre: server umount lustre-OST0001 complete [ 7918.215151] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 7928.444760] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 7969.966772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7970.220953] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7970.252431] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7970.371249] Lustre: lustre-MDT0000: new disk, initializing [ 7970.478059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7970.492596] Lustre: Skipped 1 previous similar message [ 7970.507976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7976.423225] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7989.255965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7989.435259] Lustre: 211983:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7989.506677] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7989.512092] Lustre: Skipped 1 previous similar message [ 7989.599238] Lustre: lustre-MDT0001: new disk, initializing [ 7989.716427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7989.731488] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7995.315055] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8006.441503] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8013.397839] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8020.894371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8021.216031] Lustre: 213953:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8022.773079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8022.785129] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8022.950085] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8028.011931] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8037.875685] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8043.512139] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8043.807856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8049.631287] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8050.036875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8053.739867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8053.745226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8053.764048] Lustre: Skipped 6 previous similar messages [ 8053.765427] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8053.786549] Lustre: Skipped 7 previous similar messages [ 8058.783618] Lustre: server umount lustre-OST0000 complete [ 8077.280038] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8077.814785] Lustre: server umount lustre-MDT0000 complete [ 8078.306513] LustreError: 211990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8078.320666] LustreError: 211990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 8081.994076] LustreError: 211974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401802 with bad export cookie 5907083241704522834 [ 8081.994543] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8082.028699] LustreError: 211974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8082.415526] Lustre: server umount lustre-MDT0001 complete [ 8086.407513] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 08:30:05 (1787401805) [ 8103.610257] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 8117.517694] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 8161.455272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8161.836960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8161.881366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8161.889789] Lustre: Skipped 1 previous similar message [ 8162.049310] Lustre: lustre-MDT0000: new disk, initializing [ 8162.053832] Lustre: Skipped 1 previous similar message [ 8162.275375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8167.654799] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8179.749108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8179.857734] Lustre: 218566:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8180.226329] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8186.099639] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8196.862392] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8204.367804] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8212.679981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8212.936401] Lustre: 220537:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8214.639126] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8214.924341] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8221.490921] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8231.063610] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8236.936789] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8237.248847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8242.710856] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8242.987726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8245.731268] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8245.749834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8245.753578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8245.775311] Lustre: Skipped 3 previous similar messages [ 8245.789050] Lustre: Skipped 7 previous similar messages [ 8251.629937] Lustre: server umount lustre-OST0000 complete [ 8270.308351] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8270.626962] Lustre: server umount lustre-MDT0000 complete [ 8271.343139] LustreError: 218573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8271.355890] LustreError: 218573:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8275.195343] LustreError: 218558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401995 with bad export cookie 5907083241704523709 [ 8275.201978] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8275.220842] LustreError: 218558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8275.729441] Lustre: server umount lustre-MDT0001 complete [ 8284.007448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8284.314609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8284.322359] Lustre: Skipped 5 previous similar messages [ 8290.735853] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8300.769736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8306.822872] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8314.368354] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8320.410464] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8328.057394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8336.924628] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8345.574653] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8356.324150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8356.327950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8356.341794] Lustre: Skipped 3 previous similar messages [ 8356.356570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8356.357559] Lustre: Skipped 9 previous similar messages [ 8356.373178] LustreError: Skipped 1 previous similar message [ 8362.175571] Lustre: server umount lustre-OST0000 complete [ 8372.689754] Lustre: server umount lustre-MDT0000 complete [ 8376.977871] LustreError: 222738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402097 with bad export cookie 5907083241704524598 [ 8376.992343] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8377.001731] LustreError: 222738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8377.415890] Lustre: server umount lustre-MDT0001 complete [ 8486.929905] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 8490.450476] Key type lgssc unregistered [ 8490.809310] LNet: 226421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8490.832725] LNetError: 226421:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8490.850250] LNet: Removed LNI 192.168.201.114@tcp [ 8491.962162] Key type .llcrypt unregistered [ 8491.965204] Key type ._llcrypt unregistered [ 8519.415838] Key type ._llcrypt registered [ 8519.417862] Key type .llcrypt registered [ 8519.496901] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 8537.244616] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 8538.742608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8538.802846] alg: No test for adler32 (adler32-zlib) [ 8540.047762] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 8540.388459] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 8542.183264] Key type lgssc registered [ 8543.624907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8585.674789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8585.700881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8587.082296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8587.119558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8587.186881] Lustre: lustre-MDT0000: new disk, initializing [ 8587.297345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8587.316603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8592.685495] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8604.935114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8605.145625] Lustre: 229779:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8605.228719] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8605.233624] Lustre: Skipped 1 previous similar message [ 8605.335869] Lustre: lustre-MDT0001: new disk, initializing [ 8605.479934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8605.539951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8605.555191] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8611.156901] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8621.124318] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8627.396704] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8636.083804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8636.442180] Lustre: lustre-OST0000: new disk, initializing [ 8636.457352] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8636.466088] Lustre: 231743:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8636.566765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8642.727653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8642.744322] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8642.915407] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8645.358958] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8656.988506] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8663.497430] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8663.732206] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8669.127522] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8669.482352] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8673.776309] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8673.781558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8673.795962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8673.799426] Lustre: Skipped 1 previous similar message [ 8673.822518] Lustre: Skipped 1 previous similar message [ 8677.889649] Lustre: server umount lustre-OST0000 complete [ 8682.471246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8682.482821] LustreError: Skipped 1 previous similar message [ 8682.488824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8682.515277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8684.000742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8684.004128] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8687.692139] Lustre: server umount lustre-MDT0000 complete [ 8691.865921] LustreError: 229772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402411 with bad export cookie 7754853835663509877 [ 8691.892405] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8692.351819] Lustre: server umount lustre-MDT0001 complete [ 8710.517230] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 8722.094841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8722.668737] LustreError: 234677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8722.687772] LustreError: 234677:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8722.742021] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8727.774638] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8728.044632] LustreError: 234678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8732.133251] LustreError: 234677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8736.425152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8736.834859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8741.085854] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8743.962524] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8750.894035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8751.199398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8759.238671] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8770.093842] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8770.347591] Lustre: lustre-OST0001: new disk, initializing [ 8770.350846] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8770.357220] Lustre: 236988:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8770.477199] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8777.643475] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8778.890189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8778.918218] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8779.031577] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8789.299336] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8796.761769] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8810.086761] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 08:42:09 (1787402529) [ 8813.542273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8813.556594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8813.579524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8813.584993] Lustre: Skipped 1 previous similar message [ 8818.691293] Lustre: server umount lustre-MDT0000 complete [ 8820.198485] LustreError: 236602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8820.214877] LustreError: 236602:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8822.898773] LustreError: 234657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402543 with bad export cookie 7754853835663510766 [ 8822.916513] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8822.920444] LustreError: 234657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8823.283035] Lustre: server umount lustre-MDT0001 complete [ 8837.799350] Lustre: server umount lustre-OST0000 complete [ 8841.695208] Lustre: 228001:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787402545/real 1787402545] req@ffff95e694360000 x1874227112296448/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787402561 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8841.736167] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8841.769384] Lustre: Skipped 3 previous similar messages [ 8842.460343] Lustre: server umount lustre-OST0001 complete [ 8850.263937] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 8861.917847] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 8903.246515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8903.493960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8903.548138] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8903.610421] Lustre: lustre-MDT0000: new disk, initializing [ 8903.666886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8903.679508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8908.829587] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8921.420175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8921.482937] Lustre: 240935:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8921.539167] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8921.541909] Lustre: Skipped 1 previous similar message [ 8921.603758] Lustre: lustre-MDT0001: new disk, initializing [ 8921.739846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8921.752133] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8927.078628] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8938.735434] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8945.386601] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8951.551934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8951.750205] Lustre: lustre-OST0000: new disk, initializing [ 8951.753459] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8951.759859] Lustre: 242905:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8953.839191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8953.888401] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8954.095215] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8958.507040] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8968.674694] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8973.832566] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8974.070688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8978.556809] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8978.737563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8984.548031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8984.555805] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8984.573317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8984.575608] Lustre: Skipped 4 previous similar messages [ 8986.860305] Lustre: server umount lustre-OST0000 complete [ 8993.251769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8993.263498] LustreError: Skipped 1 previous similar message [ 8993.272443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8993.283325] Lustre: Skipped 2 previous similar messages [ 8999.180132] Lustre: server umount lustre-MDT0000 complete [ 9000.417353] LustreError: 240949:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9000.441032] LustreError: 240949:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9003.059800] LustreError: 240930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402723 with bad export cookie 7754853835663512901 [ 9003.063132] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9003.071673] LustreError: 240930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9003.372677] Lustre: server umount lustre-MDT0001 complete [ 9012.110590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9012.618183] LustreError: 245122:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9012.772852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9012.782183] Lustre: Skipped 2 previous similar messages [ 9019.094349] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9028.931754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9034.730974] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9042.159942] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9048.167961] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9055.729746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9062.943099] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9070.922537] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9218.017328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9218.019837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9218.040388] Lustre: Skipped 4 previous similar messages [ 9218.072595] Lustre: Skipped 3 previous similar messages [ 9231.327165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9231.509397] Lustre: server umount lustre-OST0000 complete [ 9241.970814] Lustre: server umount lustre-MDT0000 complete [ 9243.616771] LustreError: 245122:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9243.638029] LustreError: 245122:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9245.402890] LustreError: 245102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402965 with bad export cookie 7754853835663513783 [ 9245.406491] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9245.418081] LustreError: 245102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9245.832337] Lustre: server umount lustre-MDT0001 complete [ 9342.809054] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 9345.567623] Key type lgssc unregistered [ 9345.925695] LNet: 248730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9345.934976] LNetError: 248730:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9347.001606] LNet: Removed LNI 192.168.201.114@tcp [ 9347.838274] Key type .llcrypt unregistered [ 9347.841385] Key type ._llcrypt unregistered [ 9373.520731] Key type ._llcrypt registered [ 9373.523262] Key type .llcrypt registered [ 9373.733894] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 9390.200509] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 9391.779251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9391.840107] alg: No test for adler32 (adler32-zlib) [ 9392.893635] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 9393.239807] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 9394.897781] Key type lgssc registered [ 9395.885217] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9442.013266] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 9451.859403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9451.884414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9453.251117] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9453.282194] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9453.370862] Lustre: lustre-MDT0000: new disk, initializing [ 9453.459450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9453.479361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9458.538965] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9469.051199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9469.166949] Lustre: 252801:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9469.242407] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9469.247855] Lustre: Skipped 1 previous similar message [ 9469.324850] Lustre: lustre-MDT0001: new disk, initializing [ 9469.482823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9469.527061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9469.537231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9474.932977] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9480.059636] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9486.889766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9487.214805] Lustre: lustre-OST0000: new disk, initializing [ 9487.219175] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9487.223942] Lustre: 254433:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9487.340969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9491.155757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9491.175696] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9491.354774] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9493.839977] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9503.635628] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9503.843970] Lustre: lustre-OST0001: new disk, initializing [ 9503.848658] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9503.853676] Lustre: 255303:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9503.934659] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9509.921342] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9511.488569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9511.499882] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9511.686173] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9521.280855] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9528.533497] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9541.088249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9541.094804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9541.115982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9542.630419] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9542.640057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9542.647352] Lustre: Skipped 1 previous similar message [ 9542.664176] Lustre: Skipped 3 previous similar messages [ 9547.748525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9547.753140] Lustre: Skipped 3 previous similar messages [ 9552.869708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9552.880358] Lustre: Skipped 3 previous similar messages [ 9553.375145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9553.656421] Lustre: server umount lustre-MDT0000 complete [ 9557.141811] LustreError: 252793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403277 with bad export cookie 1501872274445293023 [ 9557.146155] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9557.153417] LustreError: 252793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9557.591676] Lustre: server umount lustre-MDT0001 complete [ 9570.296848] Lustre: server umount lustre-OST0000 complete [ 9584.427823] Lustre: server umount lustre-OST0001 complete [ 9598.739671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9608.051625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9616.461092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9628.571680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9649.935926] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 9660.405162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9660.439274] Lustre: lustre-MDT0000: reset Object Index mappings [ 9660.942423] LustreError: 260317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9660.962767] LustreError: 260317:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9661.026991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9665.969799] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9666.538942] LustreError: 260318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9666.562489] LustreError: 260318:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9671.651682] LustreError: 260317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9675.846567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9675.880776] Lustre: lustre-MDT0001: reset Object Index mappings [ 9676.351260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9680.699079] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9683.289432] Lustre: 261458:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9690.092718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9696.782528] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9701.805527] LustreError: 261811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9705.083187] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9705.438993] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9705.446882] Lustre: Skipped 1 previous similar message [ 9706.521993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9706.528505] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9712.372220] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9720.163801] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9723.970795] Lustre: 263327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9731.588042] Lustre: server umount lustre-MDT0000 complete [ 9732.070328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9732.077541] LustreError: 260312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9732.082751] Lustre: Skipped 2 previous similar messages [ 9732.117833] LustreError: 260312:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9735.519565] LustreError: 260299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403455 with bad export cookie 1501872274445299813 [ 9735.533766] LustreError: 260299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9735.535259] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9736.148922] Lustre: server umount lustre-MDT0001 complete [ 9750.749985] Lustre: server umount lustre-OST0000 complete [ 9753.375093] Lustre: 250308:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787403457/real 1787403457] req@ffff95e694360a80 x1874228006743168/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787403473 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9753.406618] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9753.421607] Lustre: Skipped 2 previous similar messages [ 9754.745902] Lustre: server umount lustre-OST0001 complete [ 9774.913684] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 08:58:14 (1787403494) [ 9790.925565] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 9800.575160] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 9836.137154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9836.343851] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9836.377354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9836.462307] Lustre: lustre-MDT0000: new disk, initializing [ 9836.532352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9836.595411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9841.320654] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9852.318609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9852.459644] Lustre: 267583:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9852.511708] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9852.515702] Lustre: Skipped 1 previous similar message [ 9852.659787] Lustre: lustre-MDT0001: new disk, initializing [ 9852.864702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9852.885068] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9856.973904] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9865.765866] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9872.018582] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9878.909133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9879.238277] Lustre: lustre-OST0000: new disk, initializing [ 9879.241066] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9879.247821] Lustre: 269554:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9880.955255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9880.972572] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9881.186224] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9885.974571] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9895.952303] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9900.810134] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9901.085975] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9905.833693] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9906.056771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9911.779321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9911.785784] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9911.792649] LustreError: Skipped 1 previous similar message [ 9911.793985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9911.835784] Lustre: Skipped 1 previous similar message [ 9914.143516] Lustre: server umount lustre-OST0000 complete [ 9919.465119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9922.016561] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9922.018387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9922.033657] Lustre: Skipped 1 previous similar message [ 9922.053324] Lustre: Skipped 3 previous similar messages [ 9924.199333] Lustre: server umount lustre-MDT0000 complete [ 9927.144406] LustreError: 267595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9927.176588] LustreError: 267595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9928.086802] LustreError: 267576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403648 with bad export cookie 1501872274445301906 [ 9928.092249] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9928.103842] LustreError: 267576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9928.425236] Lustre: server umount lustre-MDT0001 complete [ 9947.711559] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 9957.823756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9958.183731] LustreError: 272498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9958.265332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9958.280970] Lustre: Skipped 2 previous similar messages [ 9962.218550] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9969.933778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9974.479692] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9977.150913] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9983.000388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9989.385622] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9998.083197] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9998.284227] Lustre: lustre-OST0001: new disk, initializing [ 9998.288125] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9998.294332] Lustre: 274806:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9999.872845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9999.888874] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10000.022701] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10004.942733] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10014.185538] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10017.557604] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10022.264908] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10022.355539] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10027.913606] Lustre: server umount lustre-MDT0000 complete [10031.077475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10031.088299] LustreError: 272495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10031.089631] Lustre: Skipped 3 previous similar messages [10031.114727] LustreError: 272495:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10031.391727] LustreError: 272480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403751 with bad export cookie 1501872274445302788 [10031.397617] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10031.402184] LustreError: 272480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10033.754494] Lustre: server umount lustre-MDT0001 complete [10047.676079] Lustre: server umount lustre-OST0000 complete [10061.127512] Lustre: server umount lustre-OST0001 complete [10076.448262] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [10085.023430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10089.610955] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10095.605876] LustreError: 277859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10095.641852] LustreError: 277859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10097.792392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10098.267789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10098.275848] Lustre: Skipped 4 previous similar messages [10102.343486] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10104.681475] Lustre: 278999:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10110.239483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10115.567210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10116.002410] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10123.489447] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10124.736718] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10129.033394] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10136.672354] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10140.286215] Lustre: 280876:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10144.733379] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10145.003456] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10153.225265] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 09:04:32 (1787403872) [10169.826812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10169.848729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10169.872666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10173.333400] Lustre: server umount lustre-MDT0000 complete [10176.957281] LustreError: 277841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403897 with bad export cookie 1501872274445309088 [10176.964294] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10176.968101] LustreError: 277841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10191.327362] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10191.826277] Lustre: server umount lustre-MDT0001 complete [10210.271363] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10210.510489] Lustre: server umount lustre-OST0000 complete [10221.063315] Lustre: server umount lustre-OST0001 complete [10228.775638] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [10240.232663] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [10289.196855] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [10300.370608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10300.720381] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10300.760751] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10300.855050] Lustre: lustre-MDT0000: new disk, initializing [10300.965681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10305.931869] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10317.030580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10317.188732] Lustre: 285124:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10317.545804] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10322.632371] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10334.818625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10335.297904] Lustre: 286757:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10337.381355] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10342.487025] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10352.858869] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10353.040842] Lustre: 287630:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10355.051515] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10355.060877] Lustre: Skipped 1 previous similar message [10355.225912] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10359.107788] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10368.870964] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10372.711765] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10372.717817] Lustre: Skipped 1 previous similar message [10388.476379] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 09:08:27 (1787404107) [10391.011499] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10391.022454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10391.030960] Lustre: Skipped 8 previous similar messages [10391.037144] Lustre: Skipped 12 previous similar messages [10393.173772] Lustre: server umount lustre-MDT0000 complete [10396.135598] LustreError: 285136:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10396.165792] LustreError: 285136:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [10397.016450] LustreError: 286756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404117 with bad export cookie 1501872274445317047 [10397.040990] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10397.041917] LustreError: 286756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10397.395430] Lustre: server umount lustre-MDT0001 complete [10411.674064] Lustre: server umount lustre-OST0000 complete [10425.936768] Lustre: server umount lustre-OST0001 complete [10433.567042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10434.294845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10434.299843] Lustre: Skipped 6 previous similar messages [10439.040393] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10447.971962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10453.542966] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10461.559948] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10468.210362] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10475.397583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10482.415828] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10490.161951] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10508.580318] loop1: detected capacity change from 0 to 204800000 [10508.842833] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10508.849972] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10508.856772] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10514.438134] loop1: detected capacity change from 0 to 204800000 [10520.894088] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10521.016204] LustreError: 289864:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs1054: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10521.042850] LustreError: 289864:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs1054 [10521.052285] LustreError: 289864:0:(mgs_handler.c:517:mgs_target_reg()) fs1054-MDT0001: Error checking target index: rc = -13 [10521.064029] LustreError: fs1054-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10521.075750] LustreError: 293595:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10521.092922] LustreError: 293595:0:(tgt_mount.c:2129:server_put_super()) no obd fs1054-MDT0001 [10521.102295] LustreError: 293595:0:(tgt_mount.c:116:server_deregister_mount()) fs1054-MDT0001 not registered [10521.177925] Lustre: server umount fs1054-MDT0001 complete [10521.182341] LustreError: 293595:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10529.807028] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10530.093717] Lustre: Setting parameter fs1054-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs1054-MDT0001 [10530.178458] Lustre: srv-fs1054-MDT0001: No data found on store. Initialize space. [10530.182481] Lustre: Skipped 4 previous similar messages [10530.292260] Lustre: fs1054-MDT0001: new disk, initializing [10530.303718] Lustre: Skipped 3 previous similar messages [10530.411956] Lustre: srv-fs1054-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10530.419322] Lustre: 294062:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1054-MDT0001-mdtlov: can't create new llog: rc = -115 [10530.428070] LustreError: 294062:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1054-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10535.365175] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10542.330475] Lustre: server umount fs1054-MDT0001 complete [10555.494234] loop1: detected capacity change from 0 to 204800000 [10560.999993] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10561.500305] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10561.866714] Lustre: 295630:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1054-MDT0001-mdtlov: can't create new llog: rc = -115 [10561.880103] LustreError: 295630:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1054-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10566.978781] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10571.833110] Lustre: server umount fs1054-MDT0001 complete [10576.358906] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10583.188941] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.14@tcp (stopping) [10583.202118] Lustre: Skipped 1 previous similar message [10586.593136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10586.614187] Lustre: Skipped 2 previous similar messages [10596.319186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10596.514402] Lustre: server umount lustre-MDT0000 complete [10605.666980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10605.860716] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10611.200437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10611.742507] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10620.706717] loop1: detected capacity change from 0 to 204800000 [10626.975572] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10627.612555] Lustre: 298079:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1054-MDT0001-mdtlov: can't create new llog: rc = -115 [10627.624205] LustreError: 298079:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1054-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10632.544300] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10637.582355] Lustre: server umount fs1054-MDT0001 complete [10652.150299] LustreError: 293055:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10652.180910] LustreError: 293055:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 105 previous similar messages [10662.385104] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10662.399331] LustreError: Skipped 1 previous similar message [10665.604346] Lustre: server umount lustre-OST0000 complete [10676.065191] Lustre: server umount lustre-MDT0000 complete [10679.870295] LustreError: 289856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404399 with bad export cookie 1501872274445319728 [10679.872949] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10679.889773] LustreError: 289856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10680.217113] Lustre: server umount lustre-MDT0001 complete [10692.081372] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [10695.853551] Key type lgssc unregistered [10696.189787] LNet: 300423:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10696.199425] LNetError: 300423:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10696.219602] LNet: Removed LNI 192.168.201.114@tcp [10697.243145] Key type .llcrypt unregistered [10697.246884] Key type ._llcrypt unregistered [10715.285996] Key type ._llcrypt registered [10715.290826] Key type .llcrypt registered [10715.440935] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [10716.857356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10716.914445] alg: No test for adler32 (adler32-zlib) [10718.030614] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [10718.391521] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [10720.079197] Key type lgssc registered [10721.138510] Lustre: Echo OBD driver; http://www.lustre.org/ [10729.175426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10729.203037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10730.858869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10735.493685] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10740.193509] LustreError: 301696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10740.210555] LustreError: 301696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10744.777581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10745.287866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10750.919389] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10759.234704] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10764.860658] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10772.146374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10772.729691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10775.798330] LustreError: 303525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10779.816336] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10781.163512] LustreError: 303709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10786.275339] LustreError: 303525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10787.641294] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10791.414918] LustreError: 303524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10791.440274] LustreError: 303524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10799.732812] LustreError: 303525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10799.767620] LustreError: 303525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10812.833242] loop1: detected capacity change from 0 to 204800000 [10813.105857] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10813.117810] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10813.140616] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10818.029655] LustreError: 303522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10818.061927] LustreError: 303522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [10818.801536] loop1: detected capacity change from 0 to 204800000 [10827.201117] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10827.407033] Lustre: fs25540-OST0000: new disk, initializing [10827.409948] Lustre: srv-fs25540-OST0000: No data found on store. Initialize space. [10827.414614] Lustre: 305433:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10827.468840] Lustre: fs25540-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10836.482543] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10845.953210] Lustre: server umount fs25540-OST0000 complete [10850.930723] LustreError: 303522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10850.960876] LustreError: 303522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [10853.871696] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10859.748864] loop1: detected capacity change from 0 to 204800000 [10864.699347] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10864.990270] Lustre: fs25540-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10872.144996] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10874.348299] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10881.444622] loop2: detected capacity change from 0 to 204800000 [10881.804973] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10881.819917] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10881.828828] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10887.065993] loop2: detected capacity change from 0 to 204800000 [10891.538726] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10891.580740] LustreError: fs25540-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10891.586148] LustreError: fs25540-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10891.591160] LustreError: 308474:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10891.594776] LustreError: 308474:0:(tgt_mount.c:2129:server_put_super()) no obd fs25540-OST0001 [10891.597694] LustreError: 308474:0:(tgt_mount.c:116:server_deregister_mount()) fs25540-OST0001 not registered [10891.625843] Lustre: server umount fs25540-OST0001 complete [10891.629176] LustreError: 308474:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10900.925384] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10901.019338] Lustre: fs25540-OST0001: new disk, initializing [10901.022639] Lustre: srv-fs25540-OST0001: No data found on store. Initialize space. [10901.027814] Lustre: 309026:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10901.075468] Lustre: fs25540-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10907.158397] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10915.297536] LustreError: 303709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10915.316974] LustreError: 303709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 37 previous similar messages [10915.624159] Lustre: server umount fs25540-OST0000 complete [10925.616984] Lustre: server umount fs25540-OST0001 complete [10938.055127] loop1: detected capacity change from 0 to 204800000 [10944.251175] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10944.476755] Lustre: fs25540-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10950.990561] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10958.075741] loop2: detected capacity change from 0 to 204800000 [10964.145469] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10970.401807] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10979.141801] Lustre: server umount fs25540-OST0000 complete [10988.428174] Lustre: server umount fs25540-OST0001 complete [11002.347212] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11002.349026] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11002.357954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11002.392243] Lustre: Skipped 1 previous similar message [11007.464527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11007.469249] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11007.476052] Lustre: Skipped 1 previous similar message [11008.197414] Lustre: server umount lustre-OST0000 complete [11012.577400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11012.599665] Lustre: Skipped 1 previous similar message [11012.605904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11012.612320] Lustre: Skipped 1 previous similar message [11017.698467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11017.719719] Lustre: Skipped 1 previous similar message [11018.040035] Lustre: server umount lustre-MDT0000 complete [11021.622830] LustreError: 301677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404741 with bad export cookie 5725335339700377582 [11021.625242] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11021.637245] LustreError: 301677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11021.919550] Lustre: server umount lustre-MDT0001 complete [11032.566743] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [11035.731913] Key type lgssc unregistered [11035.981492] LNet: 314949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11035.990115] LNetError: 314949:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11036.010839] LNet: Removed LNI 192.168.201.114@tcp [11037.053223] Key type .llcrypt unregistered [11037.057313] Key type ._llcrypt unregistered [11071.076785] Key type ._llcrypt registered [11071.083447] Key type .llcrypt registered [11071.289615] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [11086.416390] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [11087.292284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11087.449030] alg: No test for adler32 (adler32-zlib) [11088.555911] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [11088.789050] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [11090.527166] Key type lgssc registered [11092.143969] Lustre: Echo OBD driver; http://www.lustre.org/ [11137.164988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11137.202585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11138.486228] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11138.515885] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11138.582804] Lustre: lustre-MDT0000: new disk, initializing [11138.671705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11138.708418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11143.569520] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11154.505657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11154.624423] Lustre: 318826:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11154.700331] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11154.705469] Lustre: Skipped 1 previous similar message [11154.825566] Lustre: lustre-MDT0001: new disk, initializing [11154.953574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11154.999744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11155.018636] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11159.878240] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11169.818657] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11177.400908] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11185.320043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11185.543779] Lustre: lustre-OST0000: new disk, initializing [11185.552130] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11185.565399] Lustre: 320796:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11185.660395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11191.300492] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11193.901399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11193.909654] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11193.980594] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11198.842599] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11200.867833] Lustre: Mounted lustre-client [11201.666149] Lustre: Unmounted lustre-client [11211.243398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11211.266462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11211.280046] Lustre: Skipped 1 previous similar message [11216.285672] Lustre: server umount lustre-OST0000 complete [11221.472545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11221.473643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11221.475924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11221.484902] Lustre: Skipped 2 previous similar messages [11226.325106] Lustre: server umount lustre-MDT0000 complete [11230.151484] LustreError: 318817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404950 with bad export cookie 3350776542297671368 [11230.154024] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11230.172837] LustreError: 318817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11230.757159] Lustre: server umount lustre-MDT0001 complete [11243.764822] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [11247.495085] Key type lgssc unregistered [11257.887153] LNet: 1 local NIs in recovery (showing 1): 192.168.201.114@tcp [11257.952814] LNet: 322960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11257.957750] LNetError: 322960:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11330.662930] LNet: Removed LNI 192.168.201.114@tcp [11331.552134] Key type .llcrypt unregistered [11331.554445] Key type ._llcrypt unregistered [11348.822089] Key type ._llcrypt registered [11348.824394] Key type .llcrypt registered [11349.013201] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [11365.898771] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [11367.339963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11367.624227] alg: No test for adler32 (adler32-zlib) [11368.773758] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [11369.014188] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [11370.695199] Key type lgssc registered [11371.819088] Lustre: Echo OBD driver; http://www.lustre.org/ [11399.684774] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 09:25:19 (1787405119) [11406.304193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11406.331871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11407.579635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11407.614532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11407.715282] Lustre: lustre-MDT0000: new disk, initializing [11407.826692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11407.848642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11411.846252] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11421.340115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11421.478940] Lustre: 326185:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11421.542612] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11421.547692] Lustre: Skipped 1 previous similar message [11421.634030] Lustre: lustre-MDT0001: new disk, initializing [11421.731197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11421.768962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11421.785868] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11426.454157] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11436.038081] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11441.213533] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11447.056891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11447.302557] Lustre: lustre-OST0000: new disk, initializing [11447.306403] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11447.313054] Lustre: 328156:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11447.415906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11453.694377] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11455.074080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11455.089744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11455.218646] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11463.359617] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11481.070522] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11481.078539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11481.078600] LustreError: Skipped 1 previous similar message [11481.080560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11481.094016] Lustre: Skipped 1 previous similar message [11485.476521] Lustre: server umount lustre-OST0000 complete [11491.305405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11491.324624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11491.327136] Lustre: Skipped 1 previous similar message [11491.357197] Lustre: Skipped 2 previous similar messages [11495.594849] Lustre: server umount lustre-MDT0000 complete [11499.272934] LustreError: 328153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405219 with bad export cookie 3233074765799109585 [11499.285701] LustreError: 328153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11499.286376] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11499.781818] Lustre: server umount lustre-MDT0001 complete [11510.424402] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [11513.779536] Key type lgssc unregistered [11514.178782] LNet: 330064:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11514.184106] LNetError: 330064:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11515.245051] LNet: Removed LNI 192.168.201.114@tcp [11516.174593] Key type .llcrypt unregistered [11516.176602] Key type ._llcrypt unregistered [11535.648300] Key type ._llcrypt registered [11535.649582] Key type .llcrypt registered [11535.705288] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [11536.824572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11536.866707] alg: No test for adler32 (adler32-zlib) [11538.231608] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [11538.557317] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [11540.279160] Key type lgssc registered [11541.026877] Lustre: Echo OBD driver; http://www.lustre.org/ [11548.270381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11548.285982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11549.622220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11554.391785] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11559.907571] LustreError: 331338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11559.930592] LustreError: 331338:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11563.728155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11564.168109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11568.744734] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11575.740303] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11581.882466] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11589.006330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11589.353675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11595.793539] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11602.644643] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11619.899618] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11621.398655] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11625.442149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11625.453524] Lustre: Skipped 2 previous similar messages [11625.456447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11625.466203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11625.470994] Lustre: Skipped 2 previous similar messages [11629.692916] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.14@tcp (stopping) [11634.783692] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.14@tcp (stopping) [11634.795561] Lustre: Skipped 3 previous similar messages [11639.264918] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11639.550307] Lustre: server umount lustre-MDT0000 complete [11639.934947] LustreError: 331335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11643.617908] LustreError: 331320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405363 with bad export cookie 10016944371337846229 [11643.624660] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11643.637189] LustreError: 331320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11645.022805] LustreError: 331334:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11645.038663] LustreError: 331334:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11645.063405] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.14@tcp (stopping) [11645.070759] Lustre: Skipped 3 previous similar messages [11645.921814] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11648.928849] LustreError: 332579:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11648.945256] LustreError: 332579:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11650.195386] Lustre: server umount lustre-MDT0001 complete [11653.888962] Lustre: server umount lustre-OST0000 complete [11662.832946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11667.405900] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11670.197270] Lustre: 335543:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.114@tcp [11672.393986] Lustre: server umount MGS complete [11681.397741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11682.067370] LustreError: 336114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11682.117589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11687.205501] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11691.107926] LustreError: 336109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11691.120030] LustreError: 336109:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11697.117562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11697.591510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11704.484606] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11714.170510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11714.755294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11721.687794] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11731.116155] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11733.247321] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11739.762373] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11741.701258] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11747.812030] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11747.817553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11747.821038] LustreError: Skipped 1 previous similar message [11747.838715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11747.850750] Lustre: Skipped 3 previous similar messages [11748.137818] Lustre: server umount lustre-OST0000 complete [11752.928503] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11752.944544] Lustre: Skipped 1 previous similar message [11758.056500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11758.059594] Lustre: Skipped 3 previous similar messages [11758.313607] Lustre: server umount lustre-MDT0000 complete [11762.308595] LustreError: 336094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405482 with bad export cookie 10016944371337847069 [11762.318199] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11762.327570] LustreError: 336094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11762.636677] Lustre: server umount lustre-MDT0001 complete [11774.921830] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [11777.500772] Key type lgssc unregistered [11777.751695] LNet: 339554:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11777.763256] LNetError: 339554:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11777.775620] LNet: Removed LNI 192.168.201.114@tcp [11778.541121] Key type .llcrypt unregistered [11778.542924] Key type ._llcrypt unregistered [11798.038369] Key type ._llcrypt registered [11798.041239] Key type .llcrypt registered [11798.161815] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [11813.534623] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [11814.865754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11815.114609] alg: No test for adler32 (adler32-zlib) [11816.184034] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [11816.400511] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [11818.047194] Key type lgssc registered [11819.380917] Lustre: Echo OBD driver; http://www.lustre.org/ [11851.960620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11851.991243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11853.276133] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11853.311760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11853.395585] Lustre: lustre-MDT0000: new disk, initializing [11853.486864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11853.502648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11858.529889] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11868.573617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11868.737780] Lustre: 342717:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11868.831622] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11868.838426] Lustre: Skipped 1 previous similar message [11868.903633] Lustre: lustre-MDT0001: new disk, initializing [11869.084344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11869.159142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11869.165819] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11873.862432] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11883.515267] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11889.399571] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11896.082844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11896.351298] Lustre: lustre-OST0000: new disk, initializing [11896.353212] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11896.356756] Lustre: 344686:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11896.450493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11903.191290] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11905.608479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11905.627464] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11905.705416] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11916.573994] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11921.826778] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11922.089556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11926.772693] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11926.974991] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11931.620874] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11931.630452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11931.639589] LustreError: Skipped 1 previous similar message [11931.640615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11931.689453] Lustre: Skipped 1 previous similar message [11936.737154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11936.743756] Lustre: Skipped 2 previous similar messages [11941.858747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11941.872574] Lustre: Skipped 1 previous similar message [11943.391223] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11943.755378] Lustre: server umount lustre-OST0000 complete [11951.072487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11951.081276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11951.104741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11952.097837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11957.220252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11957.222966] Lustre: Skipped 2 previous similar messages [11961.311192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11961.549416] Lustre: server umount lustre-MDT0000 complete [11965.881062] LustreError: 342709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405685 with bad export cookie 8060808312494875537 [11965.884980] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11965.887900] LustreError: 342709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11966.199792] Lustre: server umount lustre-MDT0001 complete [11975.206995] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 09:34:54 (1787405694) [11992.154699] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [12002.759777] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [12044.923877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12045.210836] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12045.232715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12045.305408] Lustre: lustre-MDT0000: new disk, initializing [12045.366928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12045.394897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12050.606918] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12062.849466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12062.994791] Lustre: 349449:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12063.055080] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12063.059808] Lustre: Skipped 1 previous similar message [12063.127166] Lustre: lustre-MDT0001: new disk, initializing [12063.224883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12063.253502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12063.271200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12068.637423] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12078.962402] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12085.884692] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12093.540541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12093.766839] Lustre: lustre-OST0000: new disk, initializing [12093.770558] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12093.776092] Lustre: 351418:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12093.884549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12101.160746] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12103.334719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12103.360804] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12103.429567] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12111.570997] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12144.613038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12144.614027] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12144.664747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12144.683357] Lustre: Skipped 1 previous similar message [12149.770280] Lustre: server umount lustre-OST0000 complete [12154.852418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12154.865375] Lustre: Skipped 2 previous similar messages [12160.341143] Lustre: server umount lustre-MDT0000 complete [12164.070129] LustreError: 349456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12164.103075] LustreError: 349456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12165.046325] LustreError: 349444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405885 with bad export cookie 8060808312494876426 [12165.067585] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12165.073547] LustreError: 349444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12165.519377] Lustre: server umount lustre-MDT0001 complete [12179.666597] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [12183.197868] Key type lgssc unregistered [12183.533748] LNet: 353595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12183.544646] LNetError: 353595:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12183.568852] LNet: Removed LNI 192.168.201.114@tcp [12184.547626] Key type .llcrypt unregistered [12184.550558] Key type ._llcrypt unregistered [12278.421303] Key type ._llcrypt registered [12278.422973] Key type .llcrypt registered [12278.495124] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [12278.744509] Key type .llcrypt unregistered [12278.748278] Key type ._llcrypt unregistered [12358.913806] Key type ._llcrypt registered [12358.916487] Key type .llcrypt registered [12359.028719] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [12359.301600] Key type .llcrypt unregistered [12359.304117] Key type ._llcrypt unregistered [12394.585929] Key type ._llcrypt registered [12394.589870] Key type .llcrypt registered [12394.716689] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [12395.116806] Key type .llcrypt unregistered [12395.124440] Key type ._llcrypt unregistered [12429.100663] Key type ._llcrypt registered [12429.103014] Key type .llcrypt registered [12429.246769] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [12429.770350] Key type .llcrypt unregistered [12429.773761] Key type ._llcrypt unregistered [12517.082986] Key type ._llcrypt registered [12517.086860] Key type .llcrypt registered [12517.234570] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [12517.485651] Key type .llcrypt unregistered [12517.489568] Key type ._llcrypt unregistered [12551.021708] Key type ._llcrypt registered [12551.024198] Key type .llcrypt registered [12551.138195] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [12551.547984] Key type .llcrypt unregistered [12551.550112] Key type ._llcrypt unregistered [12582.675878] Key type ._llcrypt registered [12582.678288] Key type .llcrypt registered [12582.899672] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [12583.409413] Key type .llcrypt unregistered [12583.413629] Key type ._llcrypt unregistered [12620.095871] Key type ._llcrypt registered [12620.105243] Key type .llcrypt registered [12620.316680] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [12620.655529] Key type .llcrypt unregistered [12620.657192] Key type ._llcrypt unregistered [12655.190891] Key type ._llcrypt registered [12655.192716] Key type .llcrypt registered [12655.276774] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [12655.511453] Key type .llcrypt unregistered [12655.515178] Key type ._llcrypt unregistered [12689.450936] Key type ._llcrypt registered [12689.456511] Key type .llcrypt registered [12689.614373] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [12689.956659] Key type .llcrypt unregistered [12689.963158] Key type ._llcrypt unregistered [12759.183742] Key type ._llcrypt registered [12759.186310] Key type .llcrypt registered [12759.289244] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [12759.648625] Key type .llcrypt unregistered [12759.650314] Key type ._llcrypt unregistered [12792.234980] Key type ._llcrypt registered [12792.239747] Key type .llcrypt registered [12792.468680] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [12792.850343] Key type .llcrypt unregistered [12792.852330] Key type ._llcrypt unregistered [12826.613554] Key type ._llcrypt registered [12826.616164] Key type .llcrypt registered [12826.707279] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [12827.054383] Key type .llcrypt unregistered [12827.055963] Key type ._llcrypt unregistered [12886.004255] Key type ._llcrypt registered [12886.007512] Key type .llcrypt registered [12886.205677] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [12904.930965] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [12906.209967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12906.240714] alg: No test for adler32 (adler32-zlib) [12907.632430] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [12907.900935] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [12909.599239] Key type lgssc registered [12911.080318] Lustre: Echo OBD driver; http://www.lustre.org/ [12953.708582] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12953.737804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12955.021696] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12955.068903] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12955.166683] Lustre: tst250fs-MDT0000: new disk, initializing [12955.296794] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12955.337661] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12960.314976] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12970.598653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12970.787635] Lustre: 373952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12970.839636] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12970.846509] Lustre: Skipped 1 previous similar message [12970.908886] Lustre: tst250fs-MDT0001: new disk, initializing [12971.020323] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12971.051259] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12971.076221] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12976.490520] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12981.981416] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12989.737758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12990.204518] Lustre: tst250fs-OST0000: new disk, initializing [12990.223770] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12990.240362] Lustre: 375584:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12990.507614] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12997.293615] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12998.735034] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12998.742084] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12998.905542] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13009.083835] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13009.351628] Lustre: tst250fs-OST0001: new disk, initializing [13009.355657] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13009.360840] Lustre: 376453:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13009.512079] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13016.888331] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [13018.796377] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13018.827883] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13018.995824] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13023.784644] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13026.010118] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13026.013234] Lustre: Skipped 1 previous similar message [13049.829492] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13049.833860] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13049.862356] Lustre: Skipped 2 previous similar messages [13049.896379] Lustre: Skipped 3 previous similar messages [13054.948656] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13054.960455] Lustre: Skipped 3 previous similar messages [13057.935923] Lustre: server umount tst250fs-MDT0000 complete [13062.077025] LustreError: 373944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787406782 with bad export cookie 11635344354110860296 [13062.091128] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13062.095191] LustreError: 373944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13062.554260] Lustre: server umount tst250fs-MDT0001 complete [13078.905891] Lustre: server umount tst250fs-OST0000 complete [13081.570305] Lustre: 372175:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787406785/real 1787406785] req@ffff95e683266a00 x1874231692434816/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787406801 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13081.594853] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13081.605070] Lustre: Skipped 1 previous similar message [13082.991919] Lustre: server umount tst250fs-OST0001 complete [13104.375448] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [13114.675610] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [13151.629100] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 09:54:30 (1787406870) [13154.027176] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13174.087851] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [13191.344794] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [13227.739416] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12960 sec ========= 09:55:46 (1787406946) [13230.175739] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 09:55:48 (1787406948) === [13234.388457] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 09:55:53 (1787406953) === [13277.799680] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [13280.473237] Key type lgssc unregistered [13280.862941] LNet: 385621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13280.880631] LNetError: 385621:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13281.966225] LNet: Removed LNI 192.168.201.114@tcp [13283.204264] Key type .llcrypt unregistered [13283.205761] Key type ._llcrypt unregistered