[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 468098022 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002359] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008014] pid_max: default: 32768 minimum: 301 [ 0.009140] LSM: Security Framework initializing [ 0.010072] Yama: becoming mindful. [ 0.011041] SELinux: Initializing. [ 0.012057] *** VALIDATE selinux *** [ 0.021004] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026165] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027148] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028102] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029129] *** VALIDATE tmpfs *** [ 0.031118] *** VALIDATE proc *** [ 0.032211] *** VALIDATE cgroup *** [ 0.033008] *** VALIDATE cgroup2 *** [ 0.034265] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037043] Spectre V2 : User space: Vulnerable [ 0.038009] Speculative Store Bypass: Vulnerable [ 0.040932] debug: unmapping init [mem 0xffffffffa8259000-0xffffffffa8260fff] [ 0.042175] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043641] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044020] ... version: 2 [ 0.045016] ... bit width: 48 [ 0.046013] ... generic registers: 4 [ 0.047011] ... value mask: 0000ffffffffffff [ 0.048014] ... max period: 00007fffffffffff [ 0.048977] ... fixed-purpose events: 3 [ 0.049015] ... event mask: 000000070000000f [ 0.050315] rcu: Hierarchical SRCU implementation. [ 0.052275] smp: Bringing up secondary CPUs ... [ 0.053424] x86: Booting SMP configuration: [ 0.054029] .... node #0, CPUs: #1 #2 #3 [ 0.057373] smp: Brought up 1 node, 4 CPUs [ 0.059010] smpboot: Max logical packages: 1 [ 0.059846] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148485] node 0 deferred pages initialised in 86ms [ 0.153024] devtmpfs: initialized [ 0.154043] x86/mm: Memory block size: 128MB [ 0.157120] gcov: version magic: 0x41383552 [ 0.160313] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161060] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162283] pinctrl core: initialized pinctrl subsystem [ 0.163144] [ 0.163712] ************************************************************* [ 0.164013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165012] ** ** [ 0.166012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.167014] ** ** [ 0.168013] ** This means that this kernel is built to expose internal ** [ 0.169012] ** IOMMU data structures, which may compromise security on ** [ 0.170013] ** your system. ** [ 0.171017] ** ** [ 0.172012] ** If you see this message and you are not debugging the ** [ 0.173015] ** kernel, report this immediately to your vendor! ** [ 0.174016] ** ** [ 0.175015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.176014] ************************************************************* [ 0.177818] NET: Registered protocol family 16 [ 0.179462] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.182066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.185071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.188127] cpuidle: using governor menu [ 0.190027] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.192411] PCI: Using configuration type 1 for base access [ 0.194123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.208193] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.210016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.214029] cryptd: max_cpu_qlen set to 1000 [ 0.216264] ACPI: Added _OSI(Module Device) [ 0.218021] ACPI: Added _OSI(Processor Device) [ 0.220015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.222017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.228358] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.235295] ACPI: Interpreter enabled [ 0.237072] ACPI: PM: (supports S0 S3 S4 S5) [ 0.239016] ACPI: Using IOAPIC for interrupt routing [ 0.241134] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.245482] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.255174] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.257038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.259016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.262081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.266062] acpiphp: Slot [2] registered [ 0.267133] acpiphp: Slot [5] registered [ 0.269087] acpiphp: Slot [6] registered [ 0.270072] acpiphp: Slot [7] registered [ 0.271072] acpiphp: Slot [8] registered [ 0.272162] acpiphp: Slot [9] registered [ 0.273102] acpiphp: Slot [10] registered [ 0.274113] acpiphp: Slot [3] registered [ 0.275130] acpiphp: Slot [4] registered [ 0.275976] acpiphp: Slot [11] registered [ 0.277092] acpiphp: Slot [12] registered [ 0.278094] acpiphp: Slot [13] registered [ 0.279135] acpiphp: Slot [14] registered [ 0.281105] acpiphp: Slot [15] registered [ 0.282106] acpiphp: Slot [16] registered [ 0.284084] acpiphp: Slot [17] registered [ 0.286114] acpiphp: Slot [18] registered [ 0.288095] acpiphp: Slot [19] registered [ 0.289106] acpiphp: Slot [20] registered [ 0.291084] acpiphp: Slot [21] registered [ 0.292110] acpiphp: Slot [22] registered [ 0.294128] acpiphp: Slot [23] registered [ 0.296119] acpiphp: Slot [24] registered [ 0.297119] acpiphp: Slot [25] registered [ 0.299104] acpiphp: Slot [26] registered [ 0.300109] acpiphp: Slot [27] registered [ 0.302165] acpiphp: Slot [28] registered [ 0.304121] acpiphp: Slot [29] registered [ 0.306118] acpiphp: Slot [30] registered [ 0.308138] acpiphp: Slot [31] registered [ 0.310069] PCI host bridge to bus 0000:00 [ 0.311019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.314029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.316032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.318033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.321029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.323028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.325178] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.329107] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.332397] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.341831] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.349049] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.352023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.354020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.357021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.360419] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.363932] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.367043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.371821] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.377020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.391894] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.396966] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.403143] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.413020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.427019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.452019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.464133] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.474016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.481015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.499015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.513377] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.521017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.528016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.553022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.565653] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.576016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.587016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.614020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.624670] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.637018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.644017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.671023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.687365] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.701033] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.710022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.725023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.735744] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.738413] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.741397] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.744345] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.746239] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.750048] iommu: Default domain type: Passthrough [ 0.752191] SCSI subsystem initialized [ 0.754129] ACPI: bus type USB registered [ 0.755154] usbcore: registered new interface driver usbfs [ 0.757085] usbcore: registered new interface driver hub [ 0.759068] usbcore: registered new device driver usb [ 0.761174] pps_core: LinuxPPS API ver. 1 registered [ 0.763012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.766125] PTP clock support registered [ 0.768036] EDAC MC: Ver: 3.0.0 [ 0.769131] PCI: Using ACPI for IRQ routing [ 0.770895] NetLabel: Initializing [ 0.773016] NetLabel: domain hash size = 128 [ 0.774009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.776081] NetLabel: unlabeled traffic allowed by default [ 0.779047] vgaarb: loaded [ 0.780207] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.781015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.788000] clocksource: Switched to clocksource kvm-clock [ 0.898777] VFS: Disk quotas dquot_6.6.0 [ 0.900444] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.903347] *** VALIDATE ramfs *** [ 0.904307] *** VALIDATE hugetlbfs *** [ 0.907246] pnp: PnP ACPI init [ 0.909623] pnp: PnP ACPI: found 6 devices [ 0.930321] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.934096] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.935870] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.938504] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.941466] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.943948] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.945960] NET: Registered protocol family 2 [ 0.947924] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.951878] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.955160] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.960809] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.964194] TCP: Hash tables configured (established 65536 bind 65536) [ 0.967097] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.970178] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.972881] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.975679] NET: Registered protocol family 1 [ 0.978932] RPC: Registered named UNIX socket transport module. [ 0.981224] RPC: Registered udp transport module. [ 0.982708] RPC: Registered tcp transport module. [ 0.984328] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.986550] NET: Registered protocol family 44 [ 0.988775] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.991925] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.994557] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.997480] PCI: CLS 0 bytes, default 64 [ 0.999365] Unpacking initramfs... [ 2.396419] debug: unmapping init [mem 0xffff9a533cc54000-0xffff9a533ffbffff] [ 2.400514] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.402921] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.405984] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.894640] Initialise system trusted keyrings [ 2.896407] Key type blacklist registered [ 2.898228] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.907393] zbud: loaded [ 2.910181] *** VALIDATE nfs *** [ 2.911163] *** VALIDATE nfs4 *** [ 2.912312] pstore: using deflate compression [ 2.916096] Platform Keyring initialized [ 3.000332] NET: Registered protocol family 38 [ 3.001837] Key type asymmetric registered [ 3.003495] Asymmetric key parser 'x509' registered [ 3.005370] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.008206] io scheduler mq-deadline registered [ 3.009825] io scheduler kyber registered [ 3.011434] io scheduler bfq registered [ 3.013350] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.015907] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.018347] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.020831] ACPI: Power Button [PWRF] [ 3.025779] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.032347] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.043451] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.052076] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.065081] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.091088] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.117062] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.122693] Non-volatile memory driver v1.3 [ 3.124244] Linux agpgart interface v0.103 [ 3.153188] virtio_blk virtio1: [vda] 150608 512-byte logical blocks (77.1 MB/73.5 MiB) [ 3.155485] vda: detected capacity change from 0 to 77111296 [ 3.174861] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.177490] vdb: detected capacity change from 0 to 1073741824 [ 3.188640] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.191238] vdc: detected capacity change from 0 to 2621440000 [ 3.208244] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.211277] vdd: detected capacity change from 0 to 2621440000 [ 3.228857] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.231113] vde: detected capacity change from 0 to 4294967296 [ 3.244393] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.246766] vdf: detected capacity change from 0 to 4294967296 [ 3.253053] libphy: Fixed MDIO Bus: probed [ 3.260717] usbcore: registered new interface driver usbserial_generic [ 3.263185] usbserial: USB Serial support registered for generic [ 3.265588] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.269642] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.271494] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.273690] mousedev: PS/2 mouse device common for all mice [ 3.276122] rtc_cmos 00:05: RTC can wake from S4 [ 3.278265] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.280182] rtc_cmos 00:05: registered as rtc0 [ 3.284449] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.286121] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.288481] intel_pstate: CPU model not supported [ 3.294645] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.299348] hid: raw HID events driver (C) Jiri Kosina [ 3.301464] usbcore: registered new interface driver usbhid [ 3.303550] usbhid: USB HID core driver [ 3.305247] drop_monitor: Initializing network drop monitor service [ 3.307727] Initializing XFRM netlink socket [ 3.309811] NET: Registered protocol family 10 [ 3.312529] Segment Routing with IPv6 [ 3.313529] NET: Registered protocol family 17 [ 3.314925] mpls_gso: MPLS GSO support [ 3.319840] RAS: Correctable Errors collector initialized. [ 3.321143] AVX version of gcm_enc/dec engaged. [ 3.321968] AES CTR mode by8 optimization enabled [ 3.388392] sched_clock: Marking stable (3388363705, 0)->(4227725539, -839361834) [ 3.391471] registered taskstats version 1 [ 3.393336] Loading compiled-in X.509 certificates [ 3.394855] zswap: loaded using pool lzo/zbud [ 3.417829] Key type big_key registered [ 3.428329] Key type encrypted registered [ 3.429514] ima: No TPM chip found, activating TPM-bypass! [ 3.430973] ima: Allocated hash algorithm: sha1 [ 3.432617] ima: No architecture policies found [ 3.434353] evm: Initialising EVM extended attributes: [ 3.436344] evm: security.selinux [ 3.437548] evm: security.ima [ 3.438644] evm: security.capability [ 3.439587] evm: HMAC attrs: 0x1 [ 3.441433] rtc_cmos 00:05: setting system clock to 2026-10-02 22:13:43 UTC (1790979223) [ 3.446831] debug: unmapping init [mem 0xffffffffa9203000-0xffffffffa93fffff] [ 3.448858] debug: unmapping init [mem 0xffffffffa7f82000-0xffffffffa8258fff] [ 3.458097] Write protecting the kernel read-only data: 28672k [ 3.461492] debug: unmapping init [mem 0xffffffffa6603000-0xffffffffa67fffff] [ 3.464573] debug: unmapping init [mem 0xffffffffa6f14000-0xffffffffa6ffffff] [ 3.501894] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.509719] systemd[1]: Detected virtualization kvm. [ 3.511560] systemd[1]: Detected architecture x86-64. [ 3.513550] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.542051] systemd[1]: No hostname configured. [ 3.544135] systemd[1]: Set hostname to . [ 3.546235] random: systemd: uninitialized urandom read (16 bytes read) [ 3.548645] systemd[1]: Initializing machine ID from random generator. [ 3.595358] random: ln: uninitialized urandom read (6 bytes read) [ 3.688320] random: systemd: uninitialized urandom read (16 bytes read) [ 3.691888] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.696713] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.704987] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.238303] device-mapper: uevent: version 1.0.3 [ 4.240169] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.933119] random: fast init done [ 4.981518] scsi host0: ata_piix [ 4.996864] virtio_net virtio0 ens2: renamed from eth0 [ 5.003495] scsi host1: ata_piix [ 5.005126] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.007470] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.237358] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.944197] random: crng init done [ 9.945482] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.204889] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.269859] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.512818] SELinux: Disabled at runtime. [ 11.575682] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.581912] systemd[1]: Detected virtualization kvm. [ 11.583717] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.045827] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.049243] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.054096] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.058055] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.062246] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.069271] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.072905] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 12.221690] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ 12.460605] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.814352] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.830377] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.923363] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.941802] EDAC sbridge: Ver: 1.1.2 [ 14.462173] Key type dns_resolver registered [ 14.750514] NFS: Registering the id_resolver key type [ 14.752248] Key type id_resolver registered [ 14.753417] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg260-server login: [ 44.410437] libcfs: loading out-of-tree module taints kernel. [ 44.431703] Key type ._llcrypt registered [ 44.432864] Key type .llcrypt registered [ 44.517224] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 68.755566] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 70.324637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 70.332529] alg: No test for adler32 (adler32-zlib) [ 71.863546] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 72.476823] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 74.199262] Key type lgssc registered [ 75.494347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 103.180463] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 151.804405] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 165.556309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 165.609301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 166.849100] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 166.877421] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 167.024408] Lustre: lustre-MDT0000: new disk, initializing [ 167.116313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 167.131708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 171.771769] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.144653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 187.230751] Lustre: 6513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 187.246115] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 187.248361] Lustre: Skipped 1 previous similar message [ 187.294942] Lustre: lustre-MDT0001: new disk, initializing [ 187.328488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 187.341458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 187.346470] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 192.250473] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 198.224741] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 209.815644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 210.106600] Lustre: lustre-OST0000: new disk, initializing [ 210.109774] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 210.114505] Lustre: 8452:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 210.167872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 215.624551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 215.639571] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 215.691511] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 217.520246] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.041712] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 236.211764] Lustre: lustre-OST0001: new disk, initializing [ 236.220700] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 236.233672] Lustre: 9526:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 236.316656] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 244.593269] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 244.829107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 244.867461] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 245.060215] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 261.320444] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 270.881891] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 277.476066] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing check_logdir /tmp/testlogs/ [ 283.648702] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing yml_node [ 290.485446] Lustre: DEBUG MARKER: Client: 2.17.59.2 [ 293.413624] Lustre: DEBUG MARKER: MDS: 2.17.59.2 [ 296.472728] Lustre: DEBUG MARKER: OSS: 2.17.59.2 [ 298.303046] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 2 18:18:36 EDT 2026 [ 319.853389] Lustre: DEBUG MARKER: only running tests: 121-999 [ 321.811629] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 323.647529] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 340.967934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 340.983337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 341.012357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 342.498177] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.501080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 342.527875] Lustre: Skipped 3 previous similar messages [ 346.805390] Lustre: server umount lustre-MDT0000 complete [ 350.346044] hrtimer: interrupt took 4033604 ns [ 351.495086] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790979571 with bad export cookie 5465377209595254348 [ 351.499330] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 351.507437] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 352.739405] LustreError: 7847:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 352.740062] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 352.753468] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 352.770398] LustreError: 7847:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 352.801173] Lustre: Skipped 3 previous similar messages [ 356.772487] LustreError: 6521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 356.773522] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 356.805402] LustreError: 6521:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 356.814388] Lustre: Skipped 2 previous similar messages [ 357.926210] Lustre: server umount lustre-MDT0001 complete [ 362.439334] Lustre: server umount lustre-OST0000 complete [ 366.682126] Lustre: server umount lustre-OST0001 complete [ 374.371546] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 383.940935] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 422.866837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 423.081864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 423.108899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 423.212689] Lustre: lustre-MDT0000: new disk, initializing [ 423.304540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 423.323326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 427.915075] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 438.934909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 439.038768] Lustre: 15131:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 439.090180] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 439.094734] Lustre: Skipped 1 previous similar message [ 439.165467] Lustre: lustre-MDT0001: new disk, initializing [ 439.256076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 439.315230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 439.340934] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 443.649884] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 454.480629] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 461.025825] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 467.878902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 468.255154] Lustre: lustre-OST0000: new disk, initializing [ 468.265671] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 468.277078] Lustre: 17098:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 468.461164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 469.783080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 469.801243] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 469.990213] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 475.999527] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 487.058597] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 492.552439] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 492.800503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 499.066699] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 499.461832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 505.827766] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 505.832606] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 505.836593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 505.840545] Lustre: Skipped 1 previous similar message [ 505.857985] Lustre: Skipped 1 previous similar message [ 508.666811] Lustre: server umount lustre-OST0000 complete [ 516.065756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.066951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 516.082820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 516.092709] LustreError: Skipped 1 previous similar message [ 521.185610] LustreError: 15143:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.358770] Lustre: server umount lustre-MDT0000 complete [ 525.736666] LustreError: 15122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790979745 with bad export cookie 5465377209595255881 [ 525.739773] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 525.751744] LustreError: 15122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 526.088445] Lustre: server umount lustre-MDT0001 complete [ 532.317555] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 18:22:31 (1790979751) [ 554.002773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 554.698633] LustreError: 19976:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 554.726060] LustreError: 19976:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 554.781493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 559.795798] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 560.104074] LustreError: 19977:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.921954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 573.726479] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 582.233631] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 588.869357] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 591.295168] Lustre: Failing over lustre-MDT0000 [ 591.614727] Lustre: server umount lustre-MDT0000 complete [ 594.407470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 594.415756] LustreError: 19972:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.431921] Lustre: Skipped 2 previous similar messages [ 594.446409] LustreError: 19972:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 610.783156] Lustre: 3644:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790979814/real 1790979814] req@ffff9a5382ef3480 x1877977902258048/t0(0) o400->MGC192.168.202.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790979830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 610.812758] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 610.827073] LustreError: 19973:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 610.868719] LustreError: 19973:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 611.871554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 620.064508] LustreError: 3642:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9a5382ef0e00 x1877977902260992/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 620.349850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 620.358351] Lustre: Skipped 1 previous similar message [ 620.419029] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 625.264354] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 625.636019] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 625.650510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 625.669678] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 635.025214] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 640.165041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 640.170970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 640.189483] Lustre: Skipped 1 previous similar message [ 640.213486] Lustre: Skipped 2 previous similar messages [ 645.485872] Lustre: server umount lustre-MDT0000 complete [ 645.601331] LustreError: 19971:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 645.615461] LustreError: 19971:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 650.419369] LustreError: 19956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790979870 with bad export cookie 5465377209595257295 [ 650.449390] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 650.454836] LustreError: 19956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 651.151542] Lustre: server umount lustre-MDT0001 complete [ 660.848524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 667.333738] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 677.576294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 681.403589] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 689.773114] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 697.187568] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 699.490100] Lustre: Failing over lustre-MDT0000 [ 699.824028] Lustre: server umount lustre-MDT0000 complete [ 702.561205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 702.594876] Lustre: Skipped 1 previous similar message [ 713.188862] LustreError: 24043:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 713.212706] LustreError: 24043:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 719.327180] Lustre: 3646:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790979922/real 1790979922] req@ffff9a53c5d50e00 x1877977902288768/t0(0) o400->MGC192.168.202.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790979938 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 719.356799] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 720.051585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 730.592499] LustreError: 3642:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9a53b20bfb80 x1877977902291968/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 730.931205] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 736.260868] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 736.276304] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 736.298847] Lustre: Skipped 1 previous similar message [ 736.334406] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 736.493970] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 747.324995] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 751.589245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 751.597348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 757.756013] Lustre: server umount lustre-MDT0000 complete [ 761.846819] LustreError: 23296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790979981 with bad export cookie 5465377209595258233 [ 761.861097] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 761.867126] LustreError: 23296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 762.615329] Lustre: server umount lustre-MDT0001 complete [ 773.388527] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 18:26:32 (1790979992) [ 792.142824] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 802.532284] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 844.494659] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 855.507740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 855.761130] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 855.800083] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 855.880540] Lustre: lustre-MDT0000: new disk, initializing [ 855.943433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 855.953660] Lustre: Skipped 3 previous similar messages [ 855.976686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 861.030978] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 873.056146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 873.202527] Lustre: 29689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 873.447632] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 879.215628] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 890.256159] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 897.661443] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 907.223865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 907.554479] Lustre: 31659:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 908.995300] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 909.034420] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 909.219461] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 909.262539] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 916.469307] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 927.712582] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 933.412169] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 955.378765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 955.379449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 955.383947] LustreError: Skipped 1 previous similar message [ 955.391415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 955.391425] Lustre: Skipped 2 previous similar messages [ 955.418804] Lustre: Skipped 3 previous similar messages [ 959.753219] Lustre: server umount lustre-OST0000 complete [ 970.545312] Lustre: server umount lustre-MDT0000 complete [ 970.726569] LustreError: 29702:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 970.751196] LustreError: 29702:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 974.309875] LustreError: 29681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790980194 with bad export cookie 5465377209595258639 [ 974.323913] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 974.328516] LustreError: 29681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 974.743337] Lustre: server umount lustre-MDT0001 complete [ 984.664651] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 987.590258] Key type lgssc unregistered [ 987.863705] LNet: 33639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.868100] LNetError: 33639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.889463] LNet: Removed LNI 192.168.202.160@tcp [ 988.802163] Key type .llcrypt unregistered [ 988.803633] Key type ._llcrypt unregistered [ 1013.336546] Key type ._llcrypt registered [ 1013.340531] Key type .llcrypt registered [ 1013.502991] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 1014.680427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1014.885875] alg: No test for adler32 (adler32-zlib) [ 1016.029292] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 1016.338230] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 1018.055384] Key type lgssc registered [ 1019.477693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1028.285069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1028.314189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1030.035755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1036.248699] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1040.354161] LustreError: 35070:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1040.380708] LustreError: 35070:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1045.475811] LustreError: 35071:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1046.765961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1047.290334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1052.920485] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1061.556809] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1068.788342] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1076.152155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1076.506447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1084.179754] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1092.914937] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1093.002501] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1100.138078] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1118.331426] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 18:32:17 (1790980337) [ 1120.599771] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1123.076636] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1130.245674] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 18:32:29 (1790980349) [ 1138.671579] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 18:32:37 (1790980357) [ 1139.542688] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1147.569736] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1147.573806] Lustre: Skipped 2 previous similar messages [ 1149.426934] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 18:32:48 (1790980368) [ 1153.370801] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1153.374273] Lustre: Skipped 3 previous similar messages [ 1172.825261] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1172.832634] Lustre: Skipped 6 previous similar messages [ 1175.758650] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 18:33:14 (1790980394) [ 1178.237086] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1181.942354] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1192.598577] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 18:33:30 (1790980410) [ 1194.482113] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1194.484985] Lustre: Skipped 3 previous similar messages [ 1209.546445] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 18:33:48 (1790980428) [ 1226.214971] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1226.230758] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1226.261214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1228.257228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1228.268101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1228.270131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1228.301248] Lustre: Skipped 1 previous similar message [ 1231.226570] Lustre: server umount lustre-OST0000 complete [ 1236.964705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1236.993711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1237.023538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1242.017199] Lustre: server umount lustre-MDT0000 complete [ 1243.618617] LustreError: 35066:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1243.632360] LustreError: 35066:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1246.145439] LustreError: 35806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790980466 with bad export cookie 10138899630953219609 [ 1246.153308] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1246.156610] LustreError: 35806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1246.555985] Lustre: server umount lustre-MDT0001 complete [ 1259.302663] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 1262.733702] Key type lgssc unregistered [ 1263.168549] LNet: 42949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1263.177428] LNetError: 42949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1263.202132] LNet: Removed LNI 192.168.202.160@tcp [ 1264.682354] Key type .llcrypt unregistered [ 1264.684395] Key type ._llcrypt unregistered [ 1285.737291] Key type ._llcrypt registered [ 1285.746255] Key type .llcrypt registered [ 1285.901672] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 1301.515640] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 1303.355798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1303.423518] alg: No test for adler32 (adler32-zlib) [ 1304.664877] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 1305.151538] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 1306.871604] Key type lgssc registered [ 1308.512727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1347.946063] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1347.974110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1349.229087] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1349.283615] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1349.375320] Lustre: lustre-MDT0000: new disk, initializing [ 1349.501550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1349.518859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1354.764937] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1365.567470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1365.660188] Lustre: 46121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1365.697971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1365.701271] Lustre: Skipped 1 previous similar message [ 1365.770413] Lustre: lustre-MDT0001: new disk, initializing [ 1365.865379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1365.934479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1365.939805] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1370.665182] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1381.267643] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1387.317390] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1394.717923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1395.074761] Lustre: lustre-OST0000: new disk, initializing [ 1395.076982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1395.083046] Lustre: 48091:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1395.157295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1401.944864] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1405.026302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1405.042623] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1405.236756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1412.904369] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1418.871325] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1419.129478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1424.650824] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1424.840134] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1431.008754] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1431.013301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1431.020749] LustreError: Skipped 1 previous similar message [ 1431.022023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1431.039161] Lustre: Skipped 1 previous similar message [ 1431.061128] Lustre: Skipped 1 previous similar message [ 1436.131794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1436.149973] Lustre: Skipped 1 previous similar message [ 1441.247270] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1441.264339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1441.269161] Lustre: Skipped 1 previous similar message [ 1441.537822] Lustre: server umount lustre-OST0000 complete [ 1446.379473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.398346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1446.409862] Lustre: Skipped 1 previous similar message [ 1451.489395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1451.493985] Lustre: Skipped 1 previous similar message [ 1460.191146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1460.739037] Lustre: server umount lustre-MDT0000 complete [ 1464.749131] LustreError: 46113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790980684 with bad export cookie 10870461786241259883 [ 1464.750080] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1464.754857] LustreError: 46113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1465.110185] Lustre: server umount lustre-MDT0001 complete [ 1475.487569] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 18:38:14 (1790980694) [ 1481.884745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1482.335300] LustreError: 50555:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1482.348699] LustreError: 50555:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1482.411029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1487.681770] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1487.841027] LustreError: 50556:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1492.963571] LustreError: 50555:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1496.900549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1497.224026] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1501.692179] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1509.460414] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1516.038153] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1523.602868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1524.103444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1529.847552] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1536.824773] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1544.572917] LustreError: 53083:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1545.717337] Lustre: Modifying parameter general.timeout=1 in log params [ 1546.716080] Lustre: Modifying parameter general.timeout=2 in log params [ 1549.405501] Lustre: Modifying parameter general.timeout=5 in log params [ 1549.419608] Lustre: Skipped 2 previous similar messages [ 1553.925377] Lustre: Modifying parameter general.timeout=10 in log params [ 1553.928914] Lustre: Skipped 4 previous similar messages [ 1562.192393] Lustre: Modifying parameter general.timeout=20 in log params [ 1562.200561] Lustre: Skipped 9 previous similar messages [ 1578.358863] Lustre: Modifying parameter general.timeout=39 in log params [ 1578.365047] Lustre: Skipped 18 previous similar messages [ 1610.546995] Lustre: Modifying parameter general.timeout=71 in log params [ 1610.550675] Lustre: Skipped 31 previous similar messages [ 1674.452611] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 18:41:33 (1790980893) [ 1675.971358] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1675.973604] Lustre: Skipped 59 previous similar messages [ 1678.204691] Lustre: 60030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1680.257369] Lustre: 60128:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1682.415266] Lustre: 60243:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1686.609077] Lustre: 60439:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1686.632555] Lustre: 60439:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1718.069634] Lustre: 61907:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1718.093751] Lustre: 61907:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1718.151237] LustreError: 61907:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1723.482364] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 18:42:21 (1790980941) [ 1737.703725] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1737.711267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1737.719843] Lustre: Skipped 2 previous similar messages [ 1737.720846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1737.739744] Lustre: Skipped 4 previous similar messages [ 1741.673740] Lustre: server umount lustre-MDT0000 complete [ 1742.817845] LustreError: 50555:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1742.830497] LustreError: 50555:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1745.583661] LustreError: 50537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790980965 with bad export cookie 10870461786241260765 [ 1745.590966] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1745.593849] LustreError: 50537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1745.938398] Lustre: server umount lustre-MDT0001 complete [ 1760.599783] Lustre: server umount lustre-OST0000 complete [ 1791.703670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1792.132176] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1792.512674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1797.296421] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1807.332391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1807.410228] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1807.429880] Lustre: 64438:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1812.770539] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1820.861751] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1826.934506] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1834.864588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1835.102442] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1835.462457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1835.465513] Lustre: Skipped 1 previous similar message [ 1842.257970] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1850.803641] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1852.720731] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1852.728678] Lustre: Skipped 22 previous similar messages [ 1852.778517] Lustre: 67005:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1861.091506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1861.099773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1861.106318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1861.113713] Lustre: Skipped 2 previous similar messages [ 1861.137661] Lustre: Skipped 1 previous similar message [ 1866.827079] Lustre: server umount lustre-OST0000 complete [ 1871.333084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1877.330479] Lustre: server umount lustre-MDT0000 complete [ 1881.485608] LustreError: 64429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790981101 with bad export cookie 10870461786241263817 [ 1881.489848] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1881.493257] LustreError: 64429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1881.928718] Lustre: server umount lustre-MDT0001 complete [ 1893.485565] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 1896.264535] Key type lgssc unregistered [ 1896.536192] LNet: 68209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1896.541815] LNetError: 68209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1896.554419] LNet: Removed LNI 192.168.202.160@tcp [ 1897.374860] Key type .llcrypt unregistered [ 1897.377227] Key type ._llcrypt unregistered [ 1928.411482] Key type ._llcrypt registered [ 1928.413644] Key type .llcrypt registered [ 1928.505177] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 1929.918928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1930.002074] alg: No test for adler32 (adler32-zlib) [ 1931.145672] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 1931.473821] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 1933.247220] Key type lgssc registered [ 1934.683698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1942.534278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1942.547794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1944.004793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1948.314787] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1953.256706] LustreError: 69640:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1953.272174] LustreError: 69640:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1957.606189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1958.515241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1963.388632] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1971.135257] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1978.524337] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1986.560357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1987.258347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1994.219813] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2001.983714] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2009.360791] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2019.302861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2019.306184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2019.325511] Lustre: Skipped 1 previous similar message [ 2019.343015] Lustre: Skipped 2 previous similar messages [ 2024.160701] Lustre: server umount lustre-MDT0000 complete [ 2024.443962] LustreError: 70386:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2024.473146] LustreError: 70386:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2028.914611] LustreError: 70380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790981248 with bad export cookie 4199175872358938062 [ 2028.917908] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2028.921660] LustreError: 70380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2029.186381] Lustre: server umount lustre-MDT0001 complete [ 2045.098651] Lustre: server umount lustre-OST0000 complete [ 2074.676661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2075.034950] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2075.074558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2075.275459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2080.277053] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2091.925134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2092.145810] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2092.240596] Lustre: 74513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2092.839709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2099.094103] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2108.334756] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2114.411508] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2120.894359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2121.081995] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2121.354064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2128.128380] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2136.151737] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2137.876707] Lustre: 77091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2139.969676] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2146.794673] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2146.802326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2146.813570] Lustre: Skipped 1 previous similar message [ 2146.823607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2147.298651] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2151.204584] Lustre: server umount lustre-OST0000 complete [ 2157.540054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2157.548541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2157.568796] Lustre: Skipped 2 previous similar messages [ 2157.597048] Lustre: Skipped 3 previous similar messages [ 2161.523377] Lustre: server umount lustre-MDT0000 complete [ 2162.657347] LustreError: 75324:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2162.667687] LustreError: 75324:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2165.265366] LustreError: 74505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790981385 with bad export cookie 4199175872358938902 [ 2165.278063] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2165.284166] LustreError: 74505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2165.588171] Lustre: server umount lustre-MDT0001 complete [ 2177.276208] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 2181.077059] Key type lgssc unregistered [ 2181.469285] LNet: 78298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2181.475638] LNetError: 78298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2182.506881] LNet: Removed LNI 192.168.202.160@tcp [ 2183.953168] Key type .llcrypt unregistered [ 2183.957772] Key type ._llcrypt unregistered [ 2214.765674] Key type ._llcrypt registered [ 2214.770879] Key type .llcrypt registered [ 2214.958247] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 2216.213592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2216.258120] alg: No test for adler32 (adler32-zlib) [ 2217.661976] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 2217.925885] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 2219.647245] Key type lgssc registered [ 2221.144776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2230.701503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2230.728883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2232.417136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2237.952427] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2242.531649] LustreError: 79728:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2242.566343] LustreError: 79728:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2247.650305] LustreError: 79727:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2248.377436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2249.206409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2253.997645] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2263.809419] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2269.497587] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2278.509774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2279.337525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2286.800560] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2297.591755] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2310.094685] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2311.096797] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2313.100546] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2313.104972] Lustre: Skipped 1 previous similar message [ 2315.827282] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2315.830680] Lustre: Skipped 2 previous similar messages [ 2320.729355] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2320.739213] Lustre: Skipped 4 previous similar messages [ 2329.278167] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2329.290488] Lustre: Skipped 7 previous similar messages [ 2346.084216] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2346.090571] Lustre: Skipped 15 previous similar messages [ 2379.136262] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2379.145276] Lustre: Skipped 31 previous similar messages [ 2422.573531] Lustre: 87856:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2425.298178] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 18:54:04 (1790981644) [ 2426.907970] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2428.867024] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 18:54:07 (1790981647) [ 2436.077808] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2436.081046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2436.096901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2436.105699] Lustre: Skipped 1 previous similar message [ 2436.121132] Lustre: Skipped 1 previous similar message [ 2439.622719] Lustre: server umount lustre-OST0000 complete [ 2446.319385] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2446.335748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2446.343903] Lustre: Skipped 1 previous similar message [ 2446.382562] Lustre: Skipped 1 previous similar message [ 2450.158500] Lustre: server umount lustre-MDT0000 complete [ 2451.430620] LustreError: 79728:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2451.475326] LustreError: 79728:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2454.964949] LustreError: 79707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790981675 with bad export cookie 7374569517028793426 [ 2454.978079] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2454.991606] LustreError: 79707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2455.370491] Lustre: server umount lustre-MDT0001 complete [ 2466.951242] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 2470.193495] Key type lgssc unregistered [ 2470.529866] LNet: 89104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2470.546646] LNetError: 89104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2470.577955] LNet: Removed LNI 192.168.202.160@tcp [ 2471.535836] Key type .llcrypt unregistered [ 2471.544690] Key type ._llcrypt unregistered [ 2477.017560] Key type ._llcrypt registered [ 2477.035363] Key type .llcrypt registered [ 2477.176938] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2481.867110] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules [ 2482.513120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2482.579475] alg: No test for adler32 (adler32-zlib) [ 2484.049377] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 2484.444728] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 2486.130514] Key type lgssc registered [ 2488.091312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2489.112968] LustreError: 90320:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2509.871102] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 2512.022324] LustreError: 90320:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2518.419607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2518.457608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2519.999733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2524.878596] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2529.252641] LustreError: 91346:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2529.262747] LustreError: 91346:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2533.154752] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 18:55:52 (1790981752) [ 2534.369865] LustreError: 91347:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2536.798823] LustreError: 91378:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2536.806779] LustreError: 91378:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2536.813256] LustreError: 91378:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 2537.031891] Lustre: server umount lustre-MDT0000 complete [ 2550.995759] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 2553.889720] Key type lgssc unregistered [ 2554.406182] LNet: 92869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2554.419132] LNetError: 92869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2554.452986] LNet: Removed LNI 192.168.202.160@tcp [ 2555.878450] Key type .llcrypt unregistered [ 2555.880462] Key type ._llcrypt unregistered [ 2576.490578] Key type ._llcrypt registered [ 2576.499895] Key type .llcrypt registered [ 2576.573555] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 2577.742426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2577.890977] alg: No test for adler32 (adler32-zlib) [ 2579.129192] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 2579.508125] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 2581.303430] Key type lgssc registered [ 2582.799790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2592.210628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2592.237697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2593.697609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2599.001977] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2602.978279] LustreError: 94102:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2602.995885] LustreError: 94102:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2607.874667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2608.445753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2613.734430] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2621.389848] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2628.236971] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2635.551903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2636.401897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2643.801857] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2652.151619] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2704.775627] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 18:58:43 (1790981923) [ 2706.699795] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2708.683699] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 18:58:47 (1790981927) [ 2715.624526] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2715.625649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2715.631233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2715.654558] Lustre: Skipped 1 previous similar message [ 2719.526152] Lustre: server umount lustre-MDT0000 complete [ 2720.738625] LustreError: 94102:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2720.763168] LustreError: 94102:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2724.303167] LustreError: 94081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790981944 with bad export cookie 1547080213143694860 [ 2724.323673] LustreError: 94081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2724.326402] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2724.636212] Lustre: server umount lustre-MDT0001 complete [ 2739.756126] Lustre: server umount lustre-OST0000 complete [ 2751.767027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2752.215567] LustreError: 98399:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2752.320445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2757.290783] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2757.603939] LustreError: 98400:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2762.720957] LustreError: 98399:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2766.231134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2767.231085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2771.855063] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2779.799721] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2786.268646] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2800.247259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2800.418234] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2800.423282] LustreError: 98389:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2800.444241] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2800.449633] LustreError: 100485:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2800.454411] LustreError: 100485:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2800.458182] LustreError: 100485:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2800.508846] Lustre: server umount lustre-OST0000 complete [ 2800.511277] LustreError: 100485:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2808.786414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2808.948910] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2808.958481] LustreError: 98389:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2809.007966] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2809.018254] LustreError: 100949:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2809.027683] LustreError: 100949:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2809.035593] LustreError: 100949:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2809.130762] Lustre: server umount lustre-OST0000 complete [ 2809.134379] LustreError: 100949:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2819.149806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2819.304091] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2819.307437] Lustre: Found index 0 for lustre-OST0000, updating log [ 2819.313380] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2819.375458] Lustre: lustre-OST0000: new disk, initializing [ 2819.378992] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2819.389193] Lustre: 101482:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2819.685497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2826.141939] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2830.445672] Lustre: Failing over lustre-OST0000 [ 2830.579428] Lustre: server umount lustre-OST0000 complete [ 2836.002338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2836.024927] Lustre: Skipped 1 previous similar message [ 2836.042974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2836.049651] Lustre: Skipped 3 previous similar messages [ 2841.062884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2841.067418] Lustre: Skipped 1 previous similar message [ 2846.177027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2846.191832] Lustre: Skipped 1 previous similar message [ 2849.247163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2849.516903] Lustre: server umount lustre-MDT0000 complete [ 2851.296509] LustreError: 99646:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2851.317684] LustreError: 99646:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2852.926029] LustreError: 98381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790982072 with bad export cookie 1547080213143697604 [ 2852.931152] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2852.935696] LustreError: 98381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2853.274967] Lustre: server umount lustre-MDT0001 complete [ 2862.206263] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 19:01:21 (1790982081) [ 2880.617348] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 2890.285500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2890.664083] LustreError: 103843:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2890.755635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2895.435355] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2903.295751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2908.582880] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2910.981908] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2917.411711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2918.328503] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2926.525725] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2928.577683] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2928.593725] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2928.625498] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2933.737076] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2933.751549] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2935.892354] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2935.938710] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2936.007771] Lustre: lustre-OST0001: new disk, initializing [ 2936.010250] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2936.014671] Lustre: 106163:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2936.222175] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:52 [ 2936.256783] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2936.259216] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2936.267466] Lustre: Skipped 2 previous similar messages [ 2937.331986] Lustre: lustre-OST0000: Recovery over after 0:09, of 1 clients 1 recovered and 0 were evicted. [ 2937.334895] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2937.343327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2942.511847] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2944.147874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2944.153107] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2944.280034] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2950.743982] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2963.027441] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2970.094150] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2970.113869] Lustre: Skipped 1 previous similar message [ 2970.124897] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2970.132980] Lustre: Skipped 1 previous similar message [ 2974.152773] Lustre: server umount lustre-MDT0001 complete [ 2975.215701] LustreError: 103838:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2975.244002] LustreError: 103838:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2985.121335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2985.298714] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2985.312577] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2985.420821] Lustre: 103833:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2987.123806] LustreError: 107713:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2987.132678] LustreError: 107713:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2987.140158] LustreError: 103836:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2987.212802] LustreError: 107720:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.160@tcp: cfg command failed: rc = -17 [ 2987.234112] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2987.234112] [ 2991.598838] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2991.674713] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3002.137621] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 19:03:40 (1790982220) [ 3019.479571] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 3039.904785] Lustre: 109783:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3066.286188] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3070.156921] Lustre: 110918:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3154.913461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3154.920516] LustreError: Skipped 1 previous similar message [ 3154.925153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3154.944039] Lustre: Skipped 1 previous similar message [ 3154.959692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3154.964305] Lustre: Skipped 1 previous similar message [ 3160.545882] LustreError: 103840:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3160.561935] LustreError: 103840:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 3160.638806] Lustre: server umount lustre-MDT0000 complete [ 3164.113340] LustreError: 103823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790982384 with bad export cookie 1547080213143698605 [ 3164.124267] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3164.128265] LustreError: 103823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3164.479877] Lustre: server umount lustre-MDT0001 complete [ 3178.854278] Lustre: server umount lustre-OST0000 complete [ 3181.029129] Lustre: 93403:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790982385/real 1790982385] req@ffff9a528317fb80 x1877980531229056/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790982401 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3181.061635] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3181.080909] Lustre: Skipped 3 previous similar messages [ 3183.134445] Lustre: server umount lustre-OST0001 complete [ 3189.203876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3200.631375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3210.346474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3219.244938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3244.187122] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 3254.380127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3254.389908] Lustre: lustre-MDT0000: reset Object Index mappings [ 3254.826177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3254.839920] Lustre: Skipped 1 previous similar message [ 3259.607533] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3268.336940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3268.373398] Lustre: lustre-MDT0001: reset Object Index mappings [ 3274.112606] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3277.342923] Lustre: 116014:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3285.378108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3293.046615] LustreError: 116366:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3293.076245] LustreError: 116366:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 3293.085223] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3293.274721] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3299.401050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:386 to 0x280000401:418) [ 3302.341038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3308.026320] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3308.032349] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3308.818939] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3316.825546] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3320.943613] Lustre: 117890:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3373.037500] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 19:09:51 (1790982591) [ 3400.162140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3400.164995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3400.183555] Lustre: Skipped 3 previous similar messages [ 3400.196629] Lustre: Skipped 7 previous similar messages [ 3405.630542] Lustre: server umount lustre-MDT0000 complete [ 3410.596705] LustreError: 114846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790982630 with bad export cookie 1547080213143745659 [ 3410.619492] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3410.635230] LustreError: 114846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3417.210926] Lustre: server umount lustre-MDT0001 complete [ 3427.875140] Lustre: server umount lustre-OST0000 complete [ 3438.565289] Lustre: server umount lustre-OST0001 complete [ 3446.529797] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 3457.318210] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 3502.897406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3503.076479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3503.099896] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3503.164803] Lustre: lustre-MDT0000: new disk, initializing [ 3503.235076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3503.240605] Lustre: Skipped 3 previous similar messages [ 3503.251334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3508.544869] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3517.640671] Lustre: server umount lustre-MDT0000 complete [ 3528.257164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3528.810405] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3528.830352] Lustre: 122165:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3528.854498] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3533.475188] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3539.633433] Lustre: server umount lustre-MDT0000 complete [ 3549.495735] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 19:12:48 (1790982768) [ 3551.171140] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3553.118788] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 19:12:51 (1790982771) [ 3571.040201] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 3580.847648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3586.045220] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3595.792838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3596.003871] Lustre: 124214:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3596.058969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3596.062419] Lustre: Skipped 1 previous similar message [ 3596.141387] Lustre: lustre-MDT0001: new disk, initializing [ 3596.281593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3596.292814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3601.712099] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3607.509745] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3615.837298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3616.243395] Lustre: lustre-OST0000: new disk, initializing [ 3616.249759] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3616.268690] Lustre: 125792:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3618.065731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3618.082358] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3618.362489] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3624.783770] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3638.405415] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3638.841876] Lustre: lustre-OST0001: new disk, initializing [ 3638.848241] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3638.860510] Lustre: 126662:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3640.120538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3640.135493] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3640.332928] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3648.310171] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3658.981174] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3663.135800] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4092.653206] Lustre: 133095:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4092.664370] Lustre: 133095:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4092.678717] Lustre: 133095:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4092.698336] Lustre: 133095:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4092.708241] Lustre: 133095:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4092.713259] Lustre: 133095:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4409.299514] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:27:07 (1790983627) [ 4425.697449] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4425.707860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4425.729984] Lustre: Skipped 2 previous similar messages [ 4425.734159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4425.737972] Lustre: Skipped 6 previous similar messages [ 4428.779996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4431.201251] Lustre: server umount lustre-MDT0000 complete [ 4433.889870] LustreError: 124225:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4433.907690] LustreError: 124225:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 4435.091688] LustreError: 133110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790983655 with bad export cookie 1547080213143814000 [ 4435.094026] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4435.099679] LustreError: 133110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4439.017284] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4439.040041] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4439.045271] Lustre: Skipped 3 previous similar messages [ 4439.062163] Lustre: Skipped 5 previous similar messages [ 4441.596293] Lustre: server umount lustre-MDT0001 complete [ 4452.335533] Lustre: server umount lustre-OST0000 complete [ 4462.989458] Lustre: server umount lustre-OST0001 complete [ 4471.293466] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 4481.810541] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 4521.805590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4522.220877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4522.256390] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4522.359536] Lustre: lustre-MDT0000: new disk, initializing [ 4522.484993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4522.494801] Lustre: Skipped 5 previous similar messages [ 4522.511618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4527.721272] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4540.277174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4540.392512] Lustre: 136156:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4540.434869] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4540.438640] Lustre: Skipped 1 previous similar message [ 4540.544258] Lustre: lustre-MDT0001: new disk, initializing [ 4540.733506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4540.767571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4546.295197] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4556.961978] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4563.075993] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4570.693274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4570.980109] Lustre: lustre-OST0000: new disk, initializing [ 4570.983946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4570.992154] Lustre: 138124:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4571.115772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4571.127324] Lustre: Skipped 1 previous similar message [ 4572.395679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4572.409517] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4572.568779] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4578.661948] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4589.332139] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4599.472380] Lustre: lustre-MDD0000: changelog on [ 4602.438905] Lustre: lustre-MDD0001: changelog on [ 6679.086761] Lustre: lustre-MDD0001: changelog off [ 6681.089500] Lustre: 137032:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6682.139749] Lustre: lustre-MDD0000: changelog off [ 6690.279580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6690.301553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6690.313988] Lustre: Skipped 1 previous similar message [ 6690.319881] Lustre: Skipped 2 previous similar messages [ 6695.420317] LustreError: 139760:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6695.453343] LustreError: 139760:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6695.537379] Lustre: server umount lustre-OST0000 complete [ 6700.513658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6700.529521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6700.540135] Lustre: Skipped 1 previous similar message [ 6700.581239] Lustre: Skipped 2 previous similar messages [ 6705.633788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6705.640980] Lustre: Skipped 1 previous similar message [ 6710.752757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6710.768292] Lustre: Skipped 1 previous similar message [ 6714.335140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6714.708071] Lustre: server umount lustre-MDT0000 complete [ 6715.874035] LustreError: 136167:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6715.893791] LustreError: 136167:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6718.761564] LustreError: 136148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790985938 with bad export cookie 1547080213144638278 [ 6718.765547] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6718.770335] LustreError: 136148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6719.115056] Lustre: server umount lustre-MDT0001 complete [ 6730.483797] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 6733.577133] Key type lgssc unregistered [ 6733.987197] LNet: 147303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6733.994055] LNetError: 147303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6734.016694] LNet: Removed LNI 192.168.202.160@tcp [ 6735.191140] Key type .llcrypt unregistered [ 6735.192851] Key type ._llcrypt unregistered [ 6757.264222] Key type ._llcrypt registered [ 6757.267733] Key type .llcrypt registered [ 6757.422918] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 6773.830724] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 6774.789803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6775.083268] alg: No test for adler32 (adler32-zlib) [ 6776.140384] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 6776.395335] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 6778.079206] Key type lgssc registered [ 6779.232748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6820.983577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6821.043275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6822.365424] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6822.386587] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6822.442694] Lustre: lustre-MDT0000: new disk, initializing [ 6822.508461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6822.530509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6827.995902] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 6840.040149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6840.201099] Lustre: 150524:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6840.293936] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6840.299494] Lustre: Skipped 1 previous similar message [ 6840.432486] Lustre: lustre-MDT0001: new disk, initializing [ 6840.631829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6840.703922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6840.727860] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6846.017493] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 6855.332573] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6861.654441] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6868.518950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6868.862493] Lustre: lustre-OST0000: new disk, initializing [ 6868.865641] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6868.871358] Lustre: 152493:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6868.948571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6875.862780] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 6878.290189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6878.301977] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6878.487952] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6886.145749] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6892.873364] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_module obdecho/obdecho [ 6894.978609] LustreError: 153496:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6894.984549] LustreError: 153496:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6896.985676] LustreError: 153593:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6909.410857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6909.414207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6909.432219] Lustre: Skipped 2 previous similar messages [ 6909.446839] Lustre: Skipped 2 previous similar messages [ 6914.649739] Lustre: server umount lustre-MDT0000 complete [ 6918.458168] LustreError: 150515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790986138 with bad export cookie 13484783752162331386 [ 6918.466577] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6918.469250] LustreError: 150515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6918.940915] Lustre: server umount lustre-MDT0001 complete [ 6933.549097] Lustre: server umount lustre-OST0000 complete [ 6958.968569] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 6969.930961] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 7024.135590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7024.486954] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7024.520965] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7024.626376] Lustre: lustre-MDT0000: new disk, initializing [ 7024.738742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7024.765970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7030.681168] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7044.309935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7044.482551] Lustre: 157313:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7044.595636] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7044.604987] Lustre: Skipped 1 previous similar message [ 7044.730328] Lustre: lustre-MDT0001: new disk, initializing [ 7044.864914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7044.918334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7044.935492] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7050.827839] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7062.183611] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7069.065001] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7076.459951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7076.725335] Lustre: lustre-OST0000: new disk, initializing [ 7076.731939] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7076.742303] Lustre: 159281:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7076.845979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7085.085789] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7086.771691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7086.793693] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7087.039537] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7098.773156] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7104.887670] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7105.037516] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7111.335348] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7111.641189] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7117.795878] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7117.799198] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7117.811452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7117.820156] Lustre: Skipped 1 previous similar message [ 7117.846833] LustreError: Skipped 1 previous similar message [ 7122.930452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7122.939572] Lustre: Skipped 2 previous similar messages [ 7128.033751] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7128.037679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7128.058809] Lustre: Skipped 1 previous similar message [ 7128.319788] Lustre: server umount lustre-OST0000 complete [ 7133.152212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7133.157583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7133.171956] Lustre: Skipped 1 previous similar message [ 7139.519561] Lustre: server umount lustre-MDT0000 complete [ 7143.393227] LustreError: 158476:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7143.424734] LustreError: 158476:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 7145.126368] LustreError: 157304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790986365 with bad export cookie 13484783752162332345 [ 7145.133492] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7145.138364] LustreError: 157304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7145.463917] Lustre: server umount lustre-MDT0001 complete [ 7151.169587] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 20:12:49 (1790986369) [ 7152.926405] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7155.797965] Lustre: DEBUG MARKER: == conf-sanity test 138: MDT mount failure must not LBUG destroying recovery exports ========================================================== 20:12:53 (1790986373) [ 7179.498628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7180.030875] LustreError: 162258:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7180.108875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7185.380966] LustreError: 162259:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7185.766842] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7190.524978] LustreError: 162258:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7195.617099] LustreError: 162259:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7196.946508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7203.436549] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7212.788437] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7219.647444] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7226.024764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7226.263404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7226.268348] Lustre: Skipped 1 previous similar message [ 7234.453383] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7243.046785] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7261.363960] Lustre: *** cfs_fail_loc=718, val=0*** [ 7261.864590] Lustre: server umount lustre-MDT0000 complete [ 7264.229833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7264.245315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7264.279379] LustreError: 162259:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7267.896193] LustreError: 162239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790986487 with bad export cookie 13484783752162333227 [ 7267.899862] Lustre: *** cfs_fail_loc=718, val=0*** [ 7267.902963] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7267.913570] LustreError: 162239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7267.916437] Lustre: Skipped 3 previous similar messages [ 7268.603766] Lustre: server umount lustre-MDT0001 complete [ 7279.456103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7287.776525] Lustre: 148736:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790986491/real 1790986491] req@00000000f6b2cc12 x1877984931863936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790986507 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7287.809509] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7287.819124] Lustre: Skipped 2 previous similar messages [ 7288.799419] Lustre: 148734:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790986492/real 1790986492] req@000000006bb742db x1877984931864064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790986508 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7292.959287] Lustre: 148734:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790986496/real 1790986496] req@00000000e7197f7c x1877984931864192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790986512 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7292.960126] LustreError: 148732:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b2dab97d x1877984931865088/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7293.343809] Lustre: *** cfs_fail_loc=723, val=0*** [ 7293.345893] Lustre: Skipped 1 previous similar message [ 7293.471682] LustreError: 165638:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-122) [ 7293.486656] LustreError: 165638:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.160@tcp: cfg command failed: rc = -122 [ 7293.505951] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 7293.505951] [ 7293.534974] LustreError: MGC192.168.202.160@tcp: Confguration from log lustre-MDT0000 failed from MGS -122. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 7293.550464] LustreError: 165608:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -122 [ 7293.572810] LustreError: 165608:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -122 [ 7293.599860] LustreError: 165608:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 7293.605559] LustreError: 165608:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 7293.737871] Lustre: server umount lustre-MDT0000 complete [ 7293.744675] LustreError: 165608:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -122 [ 7297.951128] Lustre: 148736:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790986502/real 1790986502] req@00000000beaf4639 x1877984931864576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790986518 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7297.990147] Lustre: 148736:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7302.177649] Lustre: 148733:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790986506/real 1790986506] req@000000005fe1f284 x1877984931864832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790986522 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7303.667534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7303.849332] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7303.857452] LustreError: 162249:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 7304.139616] LustreError: 164091:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7304.157250] LustreError: 164091:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7304.268076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7306.280404] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7311.154261] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7323.850654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7324.860118] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7326.048235] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7329.779580] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7329.795195] Lustre: lustre-MDT0001: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 7332.374645] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7340.632857] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7342.557957] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7349.664503] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7352.012415] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7366.118345] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7366.132828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7366.158096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7366.163256] Lustre: Skipped 3 previous similar messages [ 7370.725340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7372.259377] Lustre: server umount lustre-OST0000 complete [ 7383.988211] Lustre: server umount lustre-MDT0000 complete [ 7386.081130] LustreError: 166051:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7386.105198] LustreError: 166051:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 7390.017722] LustreError: 162237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790986610 with bad export cookie 13484783752162334179 [ 7390.039218] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7390.050487] LustreError: 162237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7391.066558] Lustre: server umount lustre-MDT0001 complete [ 7404.590442] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 7408.699112] Key type lgssc unregistered [ 7408.992537] LNet: 168870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7408.999405] LNetError: 168870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7410.023393] LNet: Removed LNI 192.168.202.160@tcp [ 7410.765653] Key type .llcrypt unregistered [ 7410.767668] Key type ._llcrypt unregistered [ 7436.892640] Key type ._llcrypt registered [ 7436.895265] Key type .llcrypt registered [ 7437.096283] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 7438.799406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7438.996950] alg: No test for adler32 (adler32-zlib) [ 7440.193882] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 7440.416817] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 7442.135238] Key type lgssc registered [ 7443.948470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7453.734761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7453.771370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7455.425586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7461.324575] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7465.953917] LustreError: 170201:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7465.968477] LustreError: 170201:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7471.083645] LustreError: 170202:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7472.468922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7473.148446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7478.760500] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7487.509699] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7495.668936] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7504.053695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7504.449234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7509.833775] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7518.176445] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7526.972199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7539.694602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7539.705689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7539.737245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7540.626711] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.60@tcp (stopping) [ 7542.243724] LustreError: 170202:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7542.250332] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7542.274410] Lustre: Skipped 1 previous similar message [ 7542.311855] Lustre: server umount lustre-MDT0000 complete [ 7545.707558] LustreError: 170198:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7545.734154] LustreError: 170198:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7547.072694] LustreError: 172803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790986767 with bad export cookie 16924297827706980904 [ 7547.077626] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7547.082594] LustreError: 172803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7547.365532] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7547.400659] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7550.815495] LustreError: 170197:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7550.847775] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.60@tcp (stopping) [ 7550.851486] LustreError: 170197:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7553.789384] Lustre: server umount lustre-MDT0001 complete [ 7561.298204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7576.046587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7576.423208] LustreError: 174064:0:(llog.c:1655:llog_backup()) MGC192.168.202.160@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7576.428577] Lustre: 174064:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.160@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7576.993715] LustreError: 174069:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7577.016483] LustreError: 174069:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7593.442644] LustreError: 174071:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7593.465839] LustreError: 174071:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 7628.640615] LustreError: 174071:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7628.658460] LustreError: 174071:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 24 previous similar messages [ 7636.960210] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7636.980898] Lustre: 174064:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7677.948404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7681.747448] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7683.048552] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7690.412556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7693.150784] LustreError: 174086:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7693.182603] LustreError: 174086:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 41 previous similar messages [ 7693.292632] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xeadf2f349d63e437 [ 7693.314956] Lustre: MGC192.168.202.160@tcp: Connection restored to 0@lo (at 0@lo) [ 7693.719853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7693.781146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 7698.937809] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7700.814119] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7712.440948] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7714.294145] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7720.687361] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7722.393391] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7745.000271] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7745.014841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7745.021200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7745.021211] Lustre: Skipped 1 previous similar message [ 7745.023489] Lustre: Skipped 1 previous similar message [ 7746.327848] Lustre: server umount lustre-MDT0000 complete [ 7750.697734] LustreError: 170181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790986970 with bad export cookie 16924297827706987575 [ 7750.711494] LustreError: 170181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7750.718085] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7751.030951] Lustre: server umount lustre-MDT0001 complete [ 7765.928530] Lustre: server umount lustre-OST0000 complete [ 7789.379814] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 7800.220471] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 7841.271420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7841.654720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7841.706889] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7841.780657] Lustre: lustre-MDT0000: new disk, initializing [ 7841.844576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7841.858172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7846.550439] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7857.333063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7857.467688] Lustre: 179171:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7857.513806] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7857.517039] Lustre: Skipped 1 previous similar message [ 7857.575672] Lustre: lustre-MDT0001: new disk, initializing [ 7857.712404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7857.722685] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7863.579865] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7874.181876] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7880.647837] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7888.318136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7888.639173] Lustre: lustre-OST0000: new disk, initializing [ 7888.642359] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7888.650389] Lustre: 181140:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7888.743935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7888.748769] Lustre: Skipped 1 previous similar message [ 7895.351427] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7897.672918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7897.690063] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7897.837915] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7905.120633] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7910.929492] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7911.203437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7915.997928] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7916.205896] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7918.560787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7918.560957] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7918.577161] LustreError: Skipped 1 previous similar message [ 7918.578387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7918.578392] Lustre: Skipped 2 previous similar messages [ 7918.626414] Lustre: Skipped 2 previous similar messages [ 7932.383173] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7932.654539] Lustre: server umount lustre-OST0000 complete [ 7939.041337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7939.068065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7939.070423] Lustre: Skipped 5 previous similar messages [ 7942.899311] Lustre: server umount lustre-MDT0000 complete [ 7944.161357] LustreError: 179178:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7944.178674] LustreError: 179178:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 7946.544306] LustreError: 179164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790987166 with bad export cookie 16924297827707004641 [ 7946.558151] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7946.937815] Lustre: server umount lustre-MDT0001 complete [ 7955.455597] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 20:26:14 (1790987174) [ 7961.520828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7962.087994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7966.478576] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7975.822498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7981.365680] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7988.551596] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7994.194183] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8002.107501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8009.064550] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 8016.509427] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8030.510844] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 20:27:29 (1790987249) [ 8037.860229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8037.860946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8037.877708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8037.877716] Lustre: Skipped 1 previous similar message [ 8037.892349] Lustre: Skipped 2 previous similar messages [ 8041.668549] Lustre: server umount lustre-OST0000 complete [ 8051.652208] Lustre: server umount lustre-MDT0000 complete [ 8055.271452] LustreError: 183536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790987275 with bad export cookie 16924297827707005523 [ 8055.283922] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8055.290982] LustreError: 183536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8055.615639] Lustre: server umount lustre-MDT0001 complete [ 8066.364277] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 8068.870119] Key type lgssc unregistered [ 8069.148678] LNet: 187282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8069.164488] LNetError: 187282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8069.183855] LNet: Removed LNI 192.168.202.160@tcp [ 8069.961594] Key type .llcrypt unregistered [ 8069.963431] Key type ._llcrypt unregistered [ 8090.868139] Key type ._llcrypt registered [ 8090.869909] Key type .llcrypt registered [ 8090.989135] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 8092.234836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8092.302137] alg: No test for adler32 (adler32-zlib) [ 8093.493908] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 8093.689982] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 8095.343241] Key type lgssc registered [ 8096.515540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8104.111665] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8104.135143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8119.647807] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8124.768664] LustreError: 188540:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.160@tcp: failed processing log, type 4: rc = -110 [ 8129.888740] LustreError: 188540:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8129.899814] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8129.904225] LustreError: MGC192.168.202.160@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8129.918588] LustreError: 188540:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8129.930544] LustreError: 188540:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 8129.938531] LustreError: 188540:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 8130.034111] Lustre: server umount lustre-OST0000 complete [ 8130.037485] LustreError: 188540:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8139.327511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8140.892587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8145.670776] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 8149.987101] LustreError: 189039:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8149.997060] LustreError: 189039:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8154.313151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8154.721027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8159.030965] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 8166.899577] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8172.349465] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8179.087539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8179.228068] LustreError: 190855:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8179.237845] Lustre: 190855:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.160@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8179.387265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8185.974491] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 8192.923436] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8195.553259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8195.566306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8195.582682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8196.069030] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8196.081017] Lustre: Skipped 1 previous similar message [ 8196.094669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8196.098972] Lustre: Skipped 2 previous similar messages [ 8199.203293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8199.208880] Lustre: Skipped 2 previous similar messages [ 8204.259090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8204.265152] Lustre: Skipped 2 previous similar messages [ 8209.375701] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8209.382630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8209.410941] Lustre: Skipped 2 previous similar messages [ 8209.649851] Lustre: server umount lustre-MDT0000 complete [ 8213.235924] LustreError: 189019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790987433 with bad export cookie 739551896671012706 [ 8213.256625] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8213.559276] LustreError: 189768:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e97d84fd x1877986312745856/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8213.737722] Lustre: server umount lustre-MDT0001 complete [ 8227.152712] Lustre: server umount lustre-OST0000 complete [ 8237.757197] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 8247.380897] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 8281.955810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8282.241621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8282.278699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8282.371210] Lustre: lustre-MDT0000: new disk, initializing [ 8282.483557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8282.514462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8287.796512] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 8298.344229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8298.505730] Lustre: 194379:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8298.558952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8298.563641] Lustre: Skipped 1 previous similar message [ 8298.658758] Lustre: lustre-MDT0001: new disk, initializing [ 8298.754902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8298.792332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8298.811090] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8303.436472] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 8312.660606] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8318.568936] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8324.567586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8324.822461] Lustre: lustre-OST0000: new disk, initializing [ 8324.826810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8324.833165] Lustre: 196345:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8324.910394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8330.787883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8330.795505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8330.850055] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 8330.911827] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8340.659549] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8345.045635] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8345.231668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8349.546654] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8349.716312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8351.713558] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8351.730718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8351.746582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8351.751238] Lustre: Skipped 1 previous similar message [ 8357.533486] Lustre: server umount lustre-OST0000 complete [ 8361.954279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8361.973794] Lustre: Skipped 2 previous similar messages [ 8372.199240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8372.220507] Lustre: Skipped 7 previous similar messages [ 8375.263894] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8375.444713] Lustre: server umount lustre-MDT0000 complete [ 8377.312392] LustreError: 194390:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8377.328062] LustreError: 194390:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8379.187348] LustreError: 194369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790987599 with bad export cookie 739551896671013427 [ 8379.194122] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8379.201054] LustreError: 194369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8379.463822] Lustre: server umount lustre-MDT0001 complete [ 8387.861645] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 20:33:26 (1790987606) [ 8404.070259] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 8406.781374] Key type lgssc unregistered [ 8407.006232] LNet: 199291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8407.013896] LNetError: 199291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8407.030779] LNet: Removed LNI 192.168.202.160@tcp [ 8407.911170] Key type .llcrypt unregistered [ 8407.913398] Key type ._llcrypt unregistered [ 8425.786247] Key type ._llcrypt registered [ 8425.788743] Key type .llcrypt registered [ 8425.864416] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 8427.375764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8427.398422] alg: No test for adler32 (adler32-zlib) [ 8428.401341] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 8428.592794] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 8430.223569] Key type lgssc registered [ 8431.611530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8439.131104] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8439.143775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8440.640438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8445.566773] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 8450.018761] LustreError: 200521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8450.035475] LustreError: 200521:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8453.900592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8454.283355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8458.243402] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 8465.485500] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8471.181591] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8478.635646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8478.787936] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8478.790296] LustreError: 202385:0:(llog.c:1696:llog_backup()) MGC192.168.202.160@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8478.797595] Lustre: 202385:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.160@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8478.950748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8484.527966] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 8491.001254] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8495.072849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8495.078193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8495.091817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8495.592440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8495.605896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8495.609586] Lustre: Skipped 1 previous similar message [ 8495.625520] Lustre: Skipped 1 previous similar message [ 8500.706022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8500.714594] Lustre: Skipped 2 previous similar messages [ 8505.826234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8507.359072] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8507.474872] Lustre: server umount lustre-MDT0000 complete [ 8510.304975] LustreError: 200501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790987730 with bad export cookie 3165510413029292643 [ 8510.307387] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8510.313662] LustreError: 200501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8510.580249] Lustre: server umount lustre-MDT0001 complete [ 8523.715867] Lustre: server umount lustre-OST0000 complete [ 8532.247474] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 8540.455330] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 8569.828295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8570.015966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8570.038926] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8570.096322] Lustre: lustre-MDT0000: new disk, initializing [ 8570.152287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8570.164426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8573.886485] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 8582.822543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8582.907920] Lustre: 205907:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8582.939381] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8582.941521] Lustre: Skipped 1 previous similar message [ 8582.944304] LustreError: 206689:0:(llog.c:1696:llog_backup()) MGC192.168.202.160@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8582.948482] LustreError: 206689:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8582.955369] Lustre: 206689:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.160@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8582.974787] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8582.978632] Lustre: Skipped 1 previous similar message [ 8583.052317] Lustre: lustre-MDT0001: new disk, initializing [ 8583.130722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8583.174741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8583.183439] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8586.794761] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 8594.241220] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8599.329123] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8605.149503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8605.347818] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8605.349666] Lustre: Skipped 1 previous similar message [ 8605.351790] LustreError: 207850:0:(llog.c:1696:llog_backup()) MGC192.168.202.160@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8605.356987] LustreError: 207850:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8605.360412] Lustre: 207850:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.160@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8605.364550] Lustre: 207850:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8605.375977] Lustre: lustre-OST0000: new disk, initializing [ 8605.378069] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8605.380994] Lustre: 207874:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8605.461832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8611.469789] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 8613.433444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8613.440923] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8613.532839] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8619.654725] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8623.715206] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8623.864322] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8627.848778] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8628.057958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8634.346098] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8634.346253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8634.348195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8634.348203] Lustre: Skipped 2 previous similar messages [ 8634.352322] LustreError: Skipped 1 previous similar message [ 8634.394955] Lustre: Skipped 1 previous similar message [ 8636.183789] Lustre: server umount lustre-OST0000 complete [ 8639.456955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8639.466587] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8639.486588] Lustre: Skipped 1 previous similar message [ 8644.579898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8644.584613] Lustre: Skipped 4 previous similar messages [ 8645.401243] Lustre: server umount lustre-MDT0000 complete [ 8648.422320] LustreError: 205901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790987868 with bad export cookie 3165510413029293364 [ 8648.430198] LustreError: 205901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8648.430715] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8648.738172] Lustre: server umount lustre-MDT0001 complete [ 8655.588441] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 20:37:54 (1790987874) [ 8669.006415] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 8676.510253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8676.838645] LustreError: 211001:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8676.864749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8679.858978] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 8681.958593] LustreError: 211002:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8686.066352] LustreError: 211001:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8686.391710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8690.813273] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 8693.414214] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8698.649221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8703.351670] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 8709.380195] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8709.485175] Lustre: lustre-OST0001: new disk, initializing [ 8709.488603] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8709.493399] Lustre: 213310:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8709.564926] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8709.567846] Lustre: Skipped 2 previous similar messages [ 8713.541666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8713.557199] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8713.660688] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8714.359587] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 8723.059390] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8732.383927] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8734.289729] Lustre: DEBUG MARKER: ADD OST3 [ 8738.516838] loop: module loaded [ 8738.592762] loop0: detected capacity change from 0 to 204800000 [ 8738.753943] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8738.763528] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8738.773890] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8743.373153] loop0: detected capacity change from 0 to 204800000 [ 8745.123961] Lustre: DEBUG MARKER: STOP OST3 [ 8747.625074] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8747.772578] Lustre: lustre-OST0003: new disk, initializing [ 8747.775869] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8747.781983] Lustre: 215241:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8749.137151] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8749.285720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8749.292640] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8749.333419] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8749.392444] Lustre: Failing over lustre-OST0003 [ 8749.545508] Lustre: server umount lustre-OST0003 complete [ 8751.201439] LustreError: 215407:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8752.851802] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 8754.549226] LustreError: 212484:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8754.668165] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8754.675712] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8762.132222] Lustre: DEBUG MARKER: START OST3 again [ 8764.782656] LustreError: 213608:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8764.816619] LustreError: 213608:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 8767.671651] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8767.891829] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8769.011264] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8769.485315] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8769.486049] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8773.312489] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 8792.248827] Lustre: Failing over lustre-OST0003 [ 8792.339851] Lustre: server umount lustre-OST0003 complete [ 8793.585945] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8793.589325] LustreError: 212484:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8793.593963] Lustre: Skipped 2 previous similar messages [ 8793.605276] LustreError: 212484:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8808.946267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8808.950212] Lustre: Skipped 2 previous similar messages [ 8814.987090] Lustre: server umount lustre-MDT0000 complete [ 8817.997761] LustreError: 213309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790988038 with bad export cookie 3165510413029294246 [ 8818.001122] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8818.016351] LustreError: 213309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8819.177276] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8819.193704] Lustre: Skipped 5 previous similar messages [ 8824.589761] Lustre: server umount lustre-MDT0001 complete [ 8828.034450] Lustre: server umount lustre-OST0000 complete [ 8830.802194] Lustre: server umount lustre-OST0001 complete [ 8836.524485] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 8843.021645] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 8870.856353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8871.038833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8871.061873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8871.141660] Lustre: lustre-MDT0000: new disk, initializing [ 8871.208507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8871.210772] Lustre: Skipped 2 previous similar messages [ 8871.219492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8874.217783] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 8882.064229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8882.158985] Lustre: 220766:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8882.336495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8885.606288] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 8892.840444] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8897.223487] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8902.406727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8902.568784] Lustre: lustre-OST0000: new disk, initializing [ 8902.571288] Lustre: Skipped 1 previous similar message [ 8902.574460] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8902.577888] Lustre: Skipped 2 previous similar messages [ 8902.581725] Lustre: 222734:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8903.937442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8903.945220] Lustre: Skipped 1 previous similar message [ 8903.948811] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8904.076111] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8906.778922] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 8913.710572] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8917.318989] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8917.491729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8921.095039] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8921.260439] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8924.647676] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8924.650155] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8924.657682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8924.670942] Lustre: Skipped 8 previous similar messages [ 8928.922609] Lustre: server umount lustre-OST0000 complete [ 8933.860808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8933.865769] LustreError: Skipped 1 previous similar message [ 8937.780575] Lustre: server umount lustre-MDT0000 complete [ 8940.003318] LustreError: 220772:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8940.019123] LustreError: 220772:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 8940.660763] LustreError: 220758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790988160 with bad export cookie 3165510413029296542 [ 8940.674228] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8940.892048] Lustre: server umount lustre-MDT0001 complete [ 8943.653984] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 20:42:42 (1790988162) [ 8955.214606] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 8964.369122] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 9020.722259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9021.117754] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9021.196136] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9021.331716] Lustre: lustre-MDT0000: new disk, initializing [ 9021.480670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9021.492432] Lustre: Skipped 2 previous similar messages [ 9021.510886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9028.752524] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 9041.645881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9041.847464] Lustre: 227341:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9042.251800] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9047.796131] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 9059.014763] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9065.032267] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9074.126237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9074.633694] Lustre: 229312:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9076.299712] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9076.458589] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9082.121570] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 9094.227277] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9101.076898] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9101.448137] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9107.347613] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9107.587928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9112.545682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9112.548872] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9112.560864] Lustre: Skipped 3 previous similar messages [ 9112.568199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9112.568206] Lustre: Skipped 3 previous similar messages [ 9116.032248] Lustre: server umount lustre-OST0000 complete [ 9126.856660] Lustre: server umount lustre-MDT0000 complete [ 9127.905255] LustreError: 227353:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9127.946453] LustreError: 227353:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9130.692818] LustreError: 227335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790988350 with bad export cookie 3165510413029297424 [ 9130.700830] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9130.723074] LustreError: 227335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9131.193121] Lustre: server umount lustre-MDT0001 complete [ 9141.418115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9147.382538] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 9157.752715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9163.278127] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 9172.184356] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9178.758566] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9186.192387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9193.801381] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 9202.880727] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9213.417878] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9219.266900] Lustre: server umount lustre-OST0000 complete [ 9229.780958] Lustre: server umount lustre-MDT0000 complete [ 9234.258051] LustreError: 233377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790988454 with bad export cookie 3165510413029298299 [ 9234.260506] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9234.272532] LustreError: 233377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9234.706818] Lustre: server umount lustre-MDT0001 complete [ 9341.304764] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 9343.397183] Key type lgssc unregistered [ 9343.701865] LNet: 235192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9343.708396] LNetError: 235192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9344.743247] LNet: Removed LNI 192.168.202.160@tcp [ 9345.525377] Key type .llcrypt unregistered [ 9345.528815] Key type ._llcrypt unregistered [ 9365.979819] Key type ._llcrypt registered [ 9365.981579] Key type .llcrypt registered [ 9366.053202] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 9378.854419] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 9379.334997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9379.356642] alg: No test for adler32 (adler32-zlib) [ 9380.394265] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 9380.568687] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 9382.240744] Key type lgssc registered [ 9383.147881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9406.376199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9406.395478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9407.573518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9407.593704] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9407.640976] Lustre: lustre-MDT0000: new disk, initializing [ 9407.688061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9407.699266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9410.010590] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 9417.317084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9417.413984] Lustre: 238557:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9417.448555] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9417.451445] Lustre: Skipped 1 previous similar message [ 9417.518710] Lustre: lustre-MDT0001: new disk, initializing [ 9417.569024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9417.598026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9417.613809] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9420.261234] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 9426.130754] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9429.936476] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9434.885331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9435.076106] Lustre: lustre-OST0000: new disk, initializing [ 9435.079657] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9435.084208] Lustre: 240524:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9435.142508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9438.921146] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 9440.785131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9440.791940] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9440.835760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9445.382591] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9448.549182] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9448.682489] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9451.420666] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9451.585537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9456.102542] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9456.105617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9456.106827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9456.116790] Lustre: Skipped 1 previous similar message [ 9458.838830] Lustre: server umount lustre-OST0000 complete [ 9461.217087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9461.230301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9461.235402] Lustre: Skipped 2 previous similar messages [ 9466.338284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9466.340789] Lustre: Skipped 1 previous similar message [ 9467.444143] Lustre: server umount lustre-MDT0000 complete [ 9470.137433] LustreError: 239363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790988690 with bad export cookie 14988038369339266667 [ 9470.141382] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9470.145447] LustreError: 239363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9470.409354] Lustre: server umount lustre-MDT0001 complete [ 9482.587886] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 9488.225281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9488.455770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9491.353740] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 9493.472375] LustreError: 243458:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9493.480649] LustreError: 243458:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9496.538791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9496.724283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9499.133816] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 9500.924574] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9505.179146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9505.378141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9509.354569] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 9515.550220] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9515.674248] Lustre: lustre-OST0001: new disk, initializing [ 9515.677411] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9515.682272] Lustre: 245764:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9520.109092] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 9520.192427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9520.199705] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9520.269909] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9528.507422] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9533.401123] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9541.495933] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 20:52:40 (1790988760) [ 9543.136958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9543.141362] LustreError: Skipped 1 previous similar message [ 9543.146821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9543.152588] Lustre: Skipped 1 previous similar message [ 9543.157072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9546.212451] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9546.219533] Lustre: Skipped 2 previous similar messages [ 9549.003900] Lustre: server umount lustre-MDT0000 complete [ 9551.331610] LustreError: 244185:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9551.351611] LustreError: 244185:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9551.621733] LustreError: 243440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790988771 with bad export cookie 14988038369339267549 [ 9551.624303] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9551.626946] LustreError: 243440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9551.816749] Lustre: server umount lustre-MDT0001 complete [ 9564.709341] Lustre: server umount lustre-OST0000 complete [ 9576.302099] Lustre: server umount lustre-OST0001 complete [ 9580.661567] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 9587.938354] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 9608.722041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9608.882238] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9608.901397] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9608.961578] Lustre: lustre-MDT0000: new disk, initializing [ 9609.014947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9609.018383] Lustre: Skipped 1 previous similar message [ 9609.030267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9611.388353] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 9618.378935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9618.443156] Lustre: 249704:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9618.487371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9618.494744] Lustre: Skipped 1 previous similar message [ 9618.583471] Lustre: lustre-MDT0001: new disk, initializing [ 9618.664185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9618.672138] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9621.071968] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 9627.688378] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9632.240157] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9637.461362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9637.722305] Lustre: lustre-OST0000: new disk, initializing [ 9637.725372] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9637.730446] Lustre: 251669:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9638.954836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9638.971893] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9639.017725] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9642.308242] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 9650.297417] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9654.227773] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9654.416690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9658.276576] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9658.430357] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9659.875272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9659.882939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9659.891955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9659.894313] Lustre: Skipped 4 previous similar messages [ 9664.485880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9669.611667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9669.617129] Lustre: Skipped 3 previous similar messages [ 9674.207158] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9674.342745] Lustre: server umount lustre-OST0000 complete [ 9679.842559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9679.852395] Lustre: Skipped 1 previous similar message [ 9683.792920] Lustre: server umount lustre-MDT0000 complete [ 9684.961374] LustreError: 252133:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9684.967952] LustreError: 252133:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9686.967237] LustreError: 249695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790988907 with bad export cookie 14988038369339269684 [ 9686.979455] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9686.985301] LustreError: 249695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9687.373330] Lustre: server umount lustre-MDT0001 complete [ 9694.325417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9694.653575] LustreError: 253887:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9694.707765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9694.712410] Lustre: Skipped 2 previous similar messages [ 9697.883665] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 9699.809598] LustreError: 253888:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9703.909431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9708.080067] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 9714.694881] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9719.546458] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9724.419352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9730.111232] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 9736.192886] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9883.619731] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9883.622348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9883.627761] Lustre: Skipped 1 previous similar message [ 9883.630117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9883.632284] Lustre: Skipped 3 previous similar messages [ 9885.826624] Lustre: server umount lustre-OST0000 complete [ 9894.256036] Lustre: server umount lustre-MDT0000 complete [ 9896.283713] LustreError: 253867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790989116 with bad export cookie 14988038369339270566 [ 9896.288418] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9896.292252] LustreError: 253867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9896.500836] Lustre: server umount lustre-MDT0001 complete [10001.892112] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [10003.391689] Key type lgssc unregistered [10003.589539] LNet: 257491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10003.595125] LNetError: 257491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10003.620800] LNet: Removed LNI 192.168.202.160@tcp [10004.067208] Key type .llcrypt unregistered [10004.069677] Key type ._llcrypt unregistered [10016.745790] Key type ._llcrypt registered [10016.746861] Key type .llcrypt registered [10016.784553] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [10025.825319] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [10026.314324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10026.382381] alg: No test for adler32 (adler32-zlib) [10027.335305] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [10027.481412] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [10029.119352] Key type lgssc registered [10029.754244] Lustre: Echo OBD driver; http://www.lustre.org/ [10054.634294] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [10060.150052] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10060.177595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10061.330994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10061.351093] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10061.395355] Lustre: lustre-MDT0000: new disk, initializing [10061.450459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10061.472905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10064.076366] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10070.853762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10070.929607] Lustre: 261560:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10070.998203] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10071.003338] Lustre: Skipped 1 previous similar message [10071.052517] Lustre: lustre-MDT0001: new disk, initializing [10071.143142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10071.175092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10071.191551] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10073.754238] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10077.223286] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10080.970250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10081.122335] Lustre: lustre-OST0000: new disk, initializing [10081.125579] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10081.131137] Lustre: 263191:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10081.195211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10084.644987] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10085.929898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10085.941423] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10086.021537] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10091.043883] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10091.142644] Lustre: lustre-OST0001: new disk, initializing [10091.147602] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10091.155324] Lustre: 264066:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10091.210101] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10094.316027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10094.319154] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10094.400296] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10094.478560] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10101.889425] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10106.891219] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10115.045557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10115.048673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10115.055645] Lustre: Skipped 2 previous similar messages [10115.058365] Lustre: Skipped 1 previous similar message [10118.630796] Lustre: server umount lustre-MDT0000 complete [10120.168421] LustreError: 261568:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10120.174818] LustreError: 261568:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10120.547343] LustreError: 264064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790989340 with bad export cookie 6130954323312145933 [10120.549460] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10120.556646] LustreError: 264064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10120.699139] Lustre: server umount lustre-MDT0001 complete [10132.509217] Lustre: server umount lustre-OST0000 complete [10144.883742] Lustre: server umount lustre-OST0001 complete [10152.960402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10158.774858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10162.851639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10167.729331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10177.704510] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [10183.386741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10183.400859] Lustre: lustre-MDT0000: reset Object Index mappings [10183.644269] LustreError: 269068:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10183.702779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10187.237606] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10188.772517] LustreError: 269069:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10193.897632] LustreError: 269068:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10197.508432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10197.535236] Lustre: lustre-MDT0001: reset Object Index mappings [10198.529189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10207.519575] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10214.597776] Lustre: 270211:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10226.697212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10227.089868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10228.075274] LustreError: 270564:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10228.112163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10234.958985] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10238.441731] LustreError: 270567:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10238.478019] LustreError: 270567:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10247.507575] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10253.319061] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10255.314935] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10262.632292] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10265.996485] Lustre: 272081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10272.149399] Lustre: server umount lustre-MDT0000 complete [10273.763668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10273.776545] LustreError: 270083:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10273.777315] Lustre: Skipped 2 previous similar messages [10273.791726] LustreError: 270083:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10274.991621] LustreError: 269048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790989495 with bad export cookie 6130954323312152716 [10274.993874] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10274.997698] LustreError: 269048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10275.329509] Lustre: server umount lustre-MDT0001 complete [10288.164135] Lustre: server umount lustre-OST0000 complete [10301.715712] Lustre: server umount lustre-OST0001 complete [10318.221808] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 21:05:37 (1790989537) [10332.943631] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [10352.439936] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [10404.679592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10405.107369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10405.199066] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10405.369674] Lustre: lustre-MDT0000: new disk, initializing [10405.476444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10405.479342] Lustre: Skipped 1 previous similar message [10405.509260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10412.285500] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10425.537721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10425.635321] Lustre: 276358:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10425.671682] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10425.674848] Lustre: Skipped 1 previous similar message [10425.722294] Lustre: lustre-MDT0001: new disk, initializing [10425.884393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10425.908910] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10431.284436] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10440.969381] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10447.746296] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10455.073438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10455.473323] Lustre: lustre-OST0000: new disk, initializing [10455.481988] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10455.488967] Lustre: 278329:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10457.220595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10457.226188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10457.432493] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10462.624551] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10474.481814] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10480.263364] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10480.468329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10486.008187] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10486.346161] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10493.411724] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10493.417781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10493.418650] LustreError: Skipped 1 previous similar message [10493.420392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10493.432810] Lustre: Skipped 3 previous similar messages [10493.449053] Lustre: Skipped 3 previous similar messages [10494.903506] Lustre: server umount lustre-OST0000 complete [10502.628445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10502.643852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10502.670817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10505.612283] Lustre: server umount lustre-MDT0000 complete [10508.771996] LustreError: 276371:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10508.790993] LustreError: 276371:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10510.235532] LustreError: 276350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790989730 with bad export cookie 6130954323312154816 [10510.247273] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10510.249396] LustreError: 276350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10510.800953] Lustre: server umount lustre-MDT0001 complete [10533.552549] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [10546.537369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10547.191704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10547.200938] Lustre: Skipped 2 previous similar messages [10552.201814] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10561.996985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10567.805768] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10571.221752] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10579.181170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10585.961677] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10595.951443] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10596.156358] Lustre: lustre-OST0001: new disk, initializing [10596.159471] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10596.165353] Lustre: 283573:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10597.308993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10597.328872] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10597.430127] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10603.802970] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10613.945836] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10618.063793] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10625.398526] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10625.575471] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10644.449126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10644.463488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10644.481146] Lustre: Skipped 1 previous similar message [10644.489942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10644.494749] Lustre: Skipped 2 previous similar messages [10644.925919] Lustre: server umount lustre-MDT0000 complete [10648.559643] LustreError: 281266:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10648.597619] LustreError: 281266:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [10649.633637] LustreError: 281246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790989869 with bad export cookie 6130954323312155698 [10649.639319] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10649.640687] LustreError: 281246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10649.937532] Lustre: server umount lustre-MDT0001 complete [10665.764587] Lustre: server umount lustre-OST0000 complete [10669.027851] Lustre: 259052:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790989873/real 1790989873] req@ffff9a5281b6d880 x1877988340900992/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790989889 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10669.082078] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10669.111077] Lustre: Skipped 3 previous similar messages [10670.264791] Lustre: server umount lustre-OST0001 complete [10688.055652] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [10697.732429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10698.067442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10698.070640] Lustre: Skipped 3 previous similar messages [10702.336421] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10710.032467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10713.939391] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10716.986323] Lustre: 288067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10722.553536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10723.914795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10727.491506] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10734.436419] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10735.663422] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10739.526304] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10745.393773] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10747.955793] Lustre: 289946:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10751.664541] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10751.742152] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10761.066303] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 21:13:00 (1790989980) [10776.548091] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10776.554173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10776.555648] Lustre: Skipped 3 previous similar messages [10776.565356] Lustre: Skipped 3 previous similar messages [10780.797636] Lustre: server umount lustre-MDT0000 complete [10781.667522] LustreError: 286927:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10781.694168] LustreError: 286927:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [10783.822594] LustreError: 286906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790990003 with bad export cookie 6130954323312162551 [10783.827202] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10783.832704] LustreError: 286906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10786.787664] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10786.790921] Lustre: Skipped 1 previous similar message [10790.232092] Lustre: server umount lustre-MDT0001 complete [10799.314495] Lustre: server umount lustre-OST0000 complete [10807.737863] Lustre: server umount lustre-OST0001 complete [10811.769249] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [10817.632361] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [10849.038936] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [10857.037816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10857.234831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10857.262348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10857.310801] Lustre: lustre-MDT0000: new disk, initializing [10857.365810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10860.072745] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10867.616955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10867.714142] Lustre: 294469:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10867.958321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10870.639646] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10878.717121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10878.900126] Lustre: 296100:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10880.284400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10883.431323] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10891.058370] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10891.163667] Lustre: 296971:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10892.518280] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10895.250737] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10901.771526] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10904.403461] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10904.406668] Lustre: Skipped 1 previous similar message [10913.650567] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 21:15:32 (1790990132) [10918.369106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10918.373689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10918.375495] Lustre: Skipped 5 previous similar messages [10918.378115] Lustre: Skipped 4 previous similar messages [10921.104725] Lustre: server umount lustre-MDT0000 complete [10923.181359] LustreError: 296970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790990143 with bad export cookie 6130954323312170048 [10923.185399] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10923.188970] LustreError: 296970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10923.361536] Lustre: server umount lustre-MDT0001 complete [10935.713786] Lustre: server umount lustre-OST0000 complete [10948.370525] Lustre: server umount lustre-OST0001 complete [10952.783362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10955.132379] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10960.051184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10960.274686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10960.283098] Lustre: Skipped 8 previous similar messages [10963.130926] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10967.876168] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10971.667837] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10976.320698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10981.157923] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [10986.237981] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10996.728885] loop1: detected capacity change from 0 to 204800000 [10996.901756] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10996.908400] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10996.916542] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11000.019391] loop1: detected capacity change from 0 to 204800000 [11004.083437] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11004.132320] LustreError: 299203:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs22094: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11004.136545] LustreError: 299203:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs22094 [11004.139122] LustreError: 299203:0:(mgs_handler.c:517:mgs_target_reg()) fs22094-MDT0001: Error checking target index: rc = -13 [11004.146397] LustreError: fs22094-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11004.152939] LustreError: 302925:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [11004.158442] LustreError: 302925:0:(tgt_mount.c:2128:server_put_super()) no obd fs22094-MDT0001 [11004.162296] LustreError: 302925:0:(tgt_mount.c:116:server_deregister_mount()) fs22094-MDT0001 not registered [11004.181595] Lustre: server umount fs22094-MDT0001 complete [11004.183691] LustreError: 302925:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11009.603170] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11009.654989] Lustre: Setting parameter fs22094-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs22094-MDT0001 [11009.681950] Lustre: srv-fs22094-MDT0001: No data found on store. Initialize space. [11009.684683] Lustre: Skipped 4 previous similar messages [11009.743303] Lustre: fs22094-MDT0001: new disk, initializing [11009.745859] Lustre: Skipped 3 previous similar messages [11009.823616] Lustre: srv-fs22094-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11009.829707] Lustre: 303392:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22094-MDT0001-mdtlov: can't create new llog: rc = -115 [11009.838549] LustreError: 303392:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22094-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11012.176358] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [11016.834444] Lustre: server umount fs22094-MDT0001 complete [11023.681588] loop1: detected capacity change from 0 to 204800000 [11026.826730] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11027.011098] Lustre: 304959:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22094-MDT0001-mdtlov: can't create new llog: rc = -115 [11027.013803] LustreError: 304959:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22094-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11030.055775] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [11033.050988] Lustre: server umount fs22094-MDT0001 complete [11038.051398] LustreError: 301039:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11038.060970] LustreError: 301039:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 41 previous similar messages [11042.275408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11042.279476] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11042.283507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11042.295970] Lustre: Skipped 2 previous similar messages [11042.302819] Lustre: Skipped 3 previous similar messages [11045.257964] Lustre: server umount lustre-MDT0000 complete [11050.751197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11050.818113] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11053.653335] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [11056.106957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11058.302112] loop1: detected capacity change from 0 to 204800000 [11061.729777] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11061.993390] Lustre: 307404:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22094-MDT0001-mdtlov: can't create new llog: rc = -115 [11062.007503] LustreError: 307404:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22094-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11064.470062] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [11066.341261] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11066.965272] Lustre: server umount fs22094-MDT0001 complete [11081.699981] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11093.983138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11094.126378] Lustre: server umount lustre-OST0000 complete [11102.495692] Lustre: server umount lustre-MDT0000 complete [11105.115425] LustreError: 303937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790990325 with bad export cookie 6130954323312172729 [11105.124635] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11105.360888] Lustre: server umount lustre-MDT0001 complete [11111.976618] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [11113.834242] Key type lgssc unregistered [11114.022879] LNet: 309744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11114.028863] LNetError: 309744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11115.046621] LNet: Removed LNI 192.168.202.160@tcp [11115.564835] Key type .llcrypt unregistered [11115.567870] Key type ._llcrypt unregistered [11126.311833] Key type ._llcrypt registered [11126.313192] Key type .llcrypt registered [11126.364721] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [11126.909776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11126.920855] alg: No test for adler32 (adler32-zlib) [11127.872192] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [11127.999423] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [11129.607568] Key type lgssc registered [11130.374645] Lustre: Echo OBD driver; http://www.lustre.org/ [11135.291080] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11135.319747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11136.714197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11139.817419] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [11145.521077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11145.807198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11148.345272] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [11153.006516] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11156.324960] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11160.107919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11160.336779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11161.380749] LustreError: 312837:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11161.390842] LustreError: 312837:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11164.137046] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [11166.698776] LustreError: 312837:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11169.444932] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11170.795858] LustreError: 312835:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11170.806188] LustreError: 312835:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11175.914156] LustreError: 312836:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11175.926481] LustreError: 312836:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11181.036277] LustreError: 312836:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11181.053060] LustreError: 312836:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11182.418744] loop1: detected capacity change from 0 to 204800000 [11182.496492] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11182.506983] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11182.520181] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11186.056403] loop1: detected capacity change from 0 to 204800000 [11189.644066] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11189.736198] Lustre: fs22317-OST0000: new disk, initializing [11189.738107] Lustre: srv-fs22317-OST0000: No data found on store. Initialize space. [11189.740828] Lustre: 314740:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11189.776708] Lustre: fs22317-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11191.269074] LustreError: 312958:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11191.286894] LustreError: 312958:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11193.687551] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [11199.395212] Lustre: server umount fs22317-OST0000 complete [11206.922577] loop1: detected capacity change from 0 to 204800000 [11208.034087] LustreError: 312837:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11208.045103] LustreError: 312837:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [11210.075964] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11210.184317] Lustre: fs22317-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11213.197726] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [11217.428211] loop2: detected capacity change from 0 to 204800000 [11217.522056] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11217.527096] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11217.533525] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11220.287438] loop2: detected capacity change from 0 to 204800000 [11223.348352] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11223.410935] LustreError: fs22317-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11223.417875] LustreError: fs22317-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11223.425379] LustreError: 317777:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [11223.429202] LustreError: 317777:0:(tgt_mount.c:2128:server_put_super()) no obd fs22317-OST0001 [11223.432289] LustreError: 317777:0:(tgt_mount.c:116:server_deregister_mount()) fs22317-OST0001 not registered [11223.456039] Lustre: server umount fs22317-OST0001 complete [11223.457503] LustreError: 317777:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11230.159723] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11230.272858] Lustre: fs22317-OST0001: new disk, initializing [11230.275588] Lustre: srv-fs22317-OST0001: No data found on store. Initialize space. [11230.279317] Lustre: 318329:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11230.318203] Lustre: fs22317-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11233.294654] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [11238.579996] Lustre: server umount fs22317-OST0000 complete [11242.469045] LustreError: 312835:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11242.476743] LustreError: 312835:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [11243.865721] Lustre: server umount fs22317-OST0001 complete [11247.588127] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11251.034498] loop1: detected capacity change from 0 to 204800000 [11253.870265] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11253.965718] Lustre: fs22317-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11257.035918] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [11257.830459] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11260.672722] loop2: detected capacity change from 0 to 204800000 [11263.532881] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11266.669040] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [11271.981596] Lustre: server umount fs22317-OST0000 complete [11277.421202] Lustre: server umount fs22317-OST0001 complete [11284.448979] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11284.452337] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11284.459633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11288.544133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11288.551279] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11288.551417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11288.567537] Lustre: Skipped 1 previous similar message [11290.261598] Lustre: server umount lustre-OST0000 complete [11293.665158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11293.668354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11293.672397] Lustre: Skipped 1 previous similar message [11298.760708] Lustre: server umount lustre-MDT0000 complete [11300.987375] LustreError: 310990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790990521 with bad export cookie 14146928295369775265 [11300.989530] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11300.992383] LustreError: 310990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11301.186190] Lustre: server umount lustre-MDT0001 complete [11306.681485] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [11308.036805] Key type lgssc unregistered [11308.223336] LNet: 324242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11308.229062] LNetError: 324242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11308.247062] LNet: Removed LNI 192.168.202.160@tcp [11308.706148] Key type .llcrypt unregistered [11308.707965] Key type ._llcrypt unregistered [11325.283399] Key type ._llcrypt registered [11325.285194] Key type .llcrypt registered [11325.333794] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [11334.854939] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [11335.562554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11335.577120] alg: No test for adler32 (adler32-zlib) [11336.541354] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [11336.704083] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [11338.311247] Key type lgssc registered [11339.106584] Lustre: Echo OBD driver; http://www.lustre.org/ [11361.560473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11361.595256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11362.703595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11362.722578] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11362.770719] Lustre: lustre-MDT0000: new disk, initializing [11362.810805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11362.832764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11364.678614] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [11370.072115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11370.127066] Lustre: 328093:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11370.160133] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11370.163422] Lustre: Skipped 1 previous similar message [11370.210303] Lustre: lustre-MDT0001: new disk, initializing [11370.266911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11370.286146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11370.291852] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11372.156845] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [11377.143284] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11379.775966] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11382.867410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11382.994033] Lustre: lustre-OST0000: new disk, initializing [11382.996930] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11383.002892] Lustre: 330059:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11383.043664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11386.192489] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [11386.762874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11386.769181] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11386.830524] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11391.897339] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11393.143198] Lustre: Mounted lustre-client - version 2.17.59_2_ge06ed12 [11393.723123] Lustre: Unmounted lustre-client [11398.629618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11398.631320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11398.635727] Lustre: Skipped 1 previous similar message [11403.745946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11403.750016] Lustre: Skipped 2 previous similar messages [11404.563712] Lustre: server umount lustre-OST0000 complete [11406.303970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11406.307570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11406.312937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11408.865234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11408.865459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11412.495109] Lustre: server umount lustre-MDT0000 complete [11414.355229] LustreError: 328083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790990634 with bad export cookie 17148227090108383118 [11414.358796] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11414.359295] LustreError: 328083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11414.501389] Lustre: server umount lustre-MDT0001 complete [11419.497851] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [11420.717568] Key type lgssc unregistered [11454.689088] LNet: 332215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11454.694329] LNetError: 332215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11523.301406] LNet: Removed LNI 192.168.202.160@tcp [11523.683550] Key type .llcrypt unregistered [11523.686770] Key type ._llcrypt unregistered [11533.250217] Key type ._llcrypt registered [11533.251956] Key type .llcrypt registered [11533.320523] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [11543.160647] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [11543.674365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11543.764155] alg: No test for adler32 (adler32-zlib) [11544.731182] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [11544.895975] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [11546.514391] Key type lgssc registered [11547.209814] Lustre: Echo OBD driver; http://www.lustre.org/ [11576.765702] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 21:26:35 (1790990795) [11587.168446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11587.226320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11588.653962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11588.683554] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11588.753703] Lustre: lustre-MDT0000: new disk, initializing [11588.864937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11588.895356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11595.724176] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [11611.366567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11611.605655] Lustre: 335434:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11611.757193] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11611.767527] Lustre: Skipped 1 previous similar message [11611.952694] Lustre: lustre-MDT0001: new disk, initializing [11612.161218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11612.230991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11612.249667] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11618.332741] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [11631.264223] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11639.798879] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11646.297741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11646.546317] Lustre: lustre-OST0000: new disk, initializing [11646.549916] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11646.555878] Lustre: 337404:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11646.649080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11652.077872] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [11653.700518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11653.722698] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11653.884441] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11661.171822] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11674.602387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11674.602579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11674.608647] LustreError: Skipped 1 previous similar message [11674.622292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11674.626980] Lustre: Skipped 1 previous similar message [11679.206213] Lustre: server umount lustre-OST0000 complete [11683.812717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11683.820740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11683.834863] Lustre: Skipped 1 previous similar message [11683.841518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11684.834740] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11684.861939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11684.870881] Lustre: Skipped 1 previous similar message [11689.955353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11689.962234] Lustre: Skipped 1 previous similar message [11694.052512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11694.058630] Lustre: Skipped 1 previous similar message [11697.119224] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11697.342612] Lustre: server umount lustre-MDT0000 complete [11700.310862] LustreError: 335425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790990920 with bad export cookie 2290794045185660664 [11700.316199] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11700.326317] LustreError: 335425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11700.676364] Lustre: server umount lustre-MDT0001 complete [11710.133944] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [11712.637882] Key type lgssc unregistered [11712.856153] LNet: 339314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11712.861769] LNetError: 339314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11713.895068] LNet: Removed LNI 192.168.202.160@tcp [11714.625986] Key type .llcrypt unregistered [11714.627829] Key type ._llcrypt unregistered [11735.746410] Key type ._llcrypt registered [11735.750631] Key type .llcrypt registered [11735.917173] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [11737.248766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11737.304213] alg: No test for adler32 (adler32-zlib) [11738.282817] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [11738.572990] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [11740.319242] Key type lgssc registered [11741.397699] Lustre: Echo OBD driver; http://www.lustre.org/ [11749.250610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11749.301779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11750.939534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11755.822472] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [11760.121702] LustreError: 340606:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11760.136110] LustreError: 340606:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11763.927855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11764.198287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11768.076523] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [11775.247835] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11780.323827] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11786.278793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11786.613036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11792.446985] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [11799.586647] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11809.847623] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11811.194610] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11815.392549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11815.397894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11815.407876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11816.420701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11816.421314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11817.845529] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.60@tcp (stopping) [11817.857169] Lustre: Skipped 2 previous similar messages [11820.922880] Lustre: server umount lustre-MDT0000 complete [11821.540178] LustreError: 340602:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11821.558969] LustreError: 340602:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11822.950673] LustreError: 340601:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11824.616823] LustreError: 340586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790991044 with bad export cookie 13819974828039981393 [11824.625147] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11824.629113] LustreError: 340586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11826.657040] LustreError: 340603:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11826.657722] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11826.683061] Lustre: Skipped 1 previous similar message [11826.696071] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11831.225235] Lustre: server umount lustre-MDT0001 complete [11834.997306] Lustre: server umount lustre-OST0000 complete [11843.073907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11847.341756] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [11849.932312] Lustre: 344784:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.160@tcp [11851.889887] Lustre: server umount MGS complete [11860.096158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11860.659916] LustreError: 345355:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11860.681266] LustreError: 345355:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11860.736576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11865.758462] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [11870.186798] LustreError: 345356:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11870.219360] LustreError: 345356:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11874.241507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11874.604953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11878.637470] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [11886.152336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11886.387699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11892.963408] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [11900.198267] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11901.454839] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11906.151177] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11910.598182] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [11915.243508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11915.248665] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11915.255618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11915.258248] Lustre: Skipped 2 previous similar messages [11916.261537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11920.158995] Lustre: server umount lustre-OST0000 complete [11925.986156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11925.990685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11926.009790] Lustre: Skipped 1 previous similar message [11926.017183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11926.023099] Lustre: Skipped 2 previous similar messages [11938.271698] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11938.452243] Lustre: server umount lustre-MDT0000 complete [11941.572731] LustreError: 345335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790991161 with bad export cookie 13819974828039982233 [11941.580995] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11941.586028] LustreError: 345335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11941.928228] Lustre: server umount lustre-MDT0001 complete [11951.006138] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [11953.761481] Key type lgssc unregistered [11954.044936] LNet: 348793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11954.050735] LNetError: 348793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11954.066701] LNet: Removed LNI 192.168.202.160@tcp [11955.020915] Key type .llcrypt unregistered [11955.022554] Key type ._llcrypt unregistered [11972.004523] Key type ._llcrypt registered [11972.006210] Key type .llcrypt registered [11972.086428] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [11986.908504] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [11987.805468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11987.932981] alg: No test for adler32 (adler32-zlib) [11988.981477] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [11989.244594] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [11990.904587] Key type lgssc registered [11991.933795] Lustre: Echo OBD driver; http://www.lustre.org/ [12023.198362] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12023.247255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12024.549080] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12024.596932] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12024.708540] Lustre: lustre-MDT0000: new disk, initializing [12024.789323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12024.806585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12029.209995] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [12040.441640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12040.586171] Lustre: 351968:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12040.690405] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12040.707677] Lustre: Skipped 1 previous similar message [12040.845853] Lustre: lustre-MDT0001: new disk, initializing [12040.984551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12041.053736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12041.069769] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12046.385603] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [12057.950848] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12065.253735] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12072.671041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12073.092665] Lustre: lustre-OST0000: new disk, initializing [12073.096648] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12073.102910] Lustre: 353937:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12073.242388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12078.702861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12078.715987] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12078.846663] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12079.047764] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [12088.360176] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12092.930664] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12093.129625] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12097.159786] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12097.390243] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12099.553080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12099.565938] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12099.593354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12099.593994] Lustre: Skipped 1 previous similar message [12099.606967] Lustre: Skipped 1 previous similar message [12104.690129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12104.693880] Lustre: Skipped 1 previous similar message [12105.522285] Lustre: server umount lustre-OST0000 complete [12109.796390] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12109.820385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12114.915361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12114.926562] Lustre: Skipped 1 previous similar message [12115.669559] Lustre: server umount lustre-MDT0000 complete [12119.519940] LustreError: 351960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790991339 with bad export cookie 12779606388923147106 [12119.528951] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12119.914295] Lustre: server umount lustre-MDT0001 complete [12128.635747] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 21:35:47 (1790991347) [12144.875795] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [12153.540478] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [12185.553367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12185.784265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12185.804153] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12185.866804] Lustre: lustre-MDT0000: new disk, initializing [12185.932925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12185.952955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12189.606177] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [12199.816955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12199.961679] Lustre: 358694:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12200.008973] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12200.013236] Lustre: Skipped 1 previous similar message [12200.075995] Lustre: lustre-MDT0001: new disk, initializing [12200.163770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12200.190077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12200.197560] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12204.848361] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [12213.970830] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12218.666356] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12225.531955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12225.858745] Lustre: lustre-OST0000: new disk, initializing [12225.864475] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12225.875748] Lustre: 360662:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12226.004489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12231.995683] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [12233.756618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12233.772951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12233.861816] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12241.011976] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12269.538960] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12269.550326] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12269.553826] LustreError: Skipped 2 previous similar messages [12269.562038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12269.571490] Lustre: Skipped 2 previous similar messages [12269.585435] Lustre: Skipped 1 previous similar message [12272.424883] Lustre: server umount lustre-OST0000 complete [12277.230461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12277.238157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12279.784577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12279.790431] Lustre: Skipped 2 previous similar messages [12282.983833] Lustre: server umount lustre-MDT0000 complete [12287.256823] LustreError: 358686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790991507 with bad export cookie 12779606388923147988 [12287.258731] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12287.270575] LustreError: 358686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12287.735357] Lustre: server umount lustre-MDT0001 complete [12298.633469] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [12301.536838] Key type lgssc unregistered [12301.958444] LNet: 362813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12301.964728] LNetError: 362813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12301.993419] LNet: Removed LNI 192.168.202.160@tcp [12302.765610] Key type .llcrypt unregistered [12302.767448] Key type ._llcrypt unregistered [12321.126944] Key type ._llcrypt registered [12321.129104] Key type .llcrypt registered [12321.204849] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [12322.172775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12322.308427] alg: No test for adler32 (adler32-zlib) [12323.276634] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [12323.517695] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [12325.207254] Key type lgssc registered [12326.192541] Lustre: Echo OBD driver; http://www.lustre.org/ [12332.635137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12332.644507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12333.971183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12338.647262] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [12344.300044] LustreError: 364103:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12344.311669] LustreError: 364103:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12347.284445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12347.654875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12352.197938] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [12359.501534] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12365.689914] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12372.559494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12373.027684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12378.745342] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [12385.498689] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12430.306273] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12430.313288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12430.329320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12433.376327] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12433.379053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12433.380502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12433.397771] Lustre: Skipped 1 previous similar message [12435.196122] Lustre: server umount lustre-OST0000 complete [12438.497710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12438.509663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12438.517258] Lustre: Skipped 1 previous similar message [12443.619239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12444.909676] Lustre: server umount lustre-MDT0000 complete [12448.636651] LustreError: 364084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790991668 with bad export cookie 15322295052497458503 [12448.644460] LustreError: 364084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12448.652336] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12448.977975] Lustre: server umount lustre-MDT0001 complete [12458.603859] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [12461.119944] Key type lgssc unregistered [12461.406797] LNet: 369706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12461.412168] LNetError: 369706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12462.441872] LNet: Removed LNI 192.168.202.160@tcp [12463.106936] Key type .llcrypt unregistered [12463.109161] Key type ._llcrypt unregistered [12536.957876] Key type ._llcrypt registered [12536.968473] Key type .llcrypt registered [12537.047583] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [12537.260509] Key type .llcrypt unregistered [12537.262693] Key type ._llcrypt unregistered [12596.524713] Key type ._llcrypt registered [12596.527573] Key type .llcrypt registered [12596.606808] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [12596.777409] Key type .llcrypt unregistered [12596.780173] Key type ._llcrypt unregistered [12620.339720] Key type ._llcrypt registered [12620.343782] Key type .llcrypt registered [12620.464707] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [12620.717805] Key type .llcrypt unregistered [12620.722392] Key type ._llcrypt unregistered [12645.301747] Key type ._llcrypt registered [12645.303752] Key type .llcrypt registered [12645.390484] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [12645.667404] Key type .llcrypt unregistered [12645.668732] Key type ._llcrypt unregistered [12713.334864] Key type ._llcrypt registered [12713.336711] Key type .llcrypt registered [12713.442255] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [12713.673521] Key type .llcrypt unregistered [12713.680073] Key type ._llcrypt unregistered [12737.197878] Key type ._llcrypt registered [12737.199757] Key type .llcrypt registered [12737.292496] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [12737.558601] Key type .llcrypt unregistered [12737.561178] Key type ._llcrypt unregistered [12763.573962] Key type ._llcrypt registered [12763.577700] Key type .llcrypt registered [12763.699462] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [12763.961601] Key type .llcrypt unregistered [12763.963245] Key type ._llcrypt unregistered [12789.176806] Key type ._llcrypt registered [12789.182781] Key type .llcrypt registered [12789.266322] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [12789.497512] Key type .llcrypt unregistered [12789.499727] Key type ._llcrypt unregistered [12811.314657] Key type ._llcrypt registered [12811.316618] Key type .llcrypt registered [12811.394461] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [12811.636392] Key type .llcrypt unregistered [12811.638981] Key type ._llcrypt unregistered [12834.074456] Key type ._llcrypt registered [12834.075660] Key type .llcrypt registered [12834.139774] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [12834.301412] Key type .llcrypt unregistered [12834.305934] Key type ._llcrypt unregistered [12880.524229] Key type ._llcrypt registered [12880.526501] Key type .llcrypt registered [12880.611777] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [12880.830338] Key type .llcrypt unregistered [12880.831945] Key type ._llcrypt unregistered [12900.697073] Key type ._llcrypt registered [12900.700312] Key type .llcrypt registered [12900.777390] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [12900.979429] Key type .llcrypt unregistered [12900.980820] Key type ._llcrypt unregistered [12924.696667] Key type ._llcrypt registered [12924.698825] Key type .llcrypt registered [12924.797593] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [12924.990257] Key type .llcrypt unregistered [12924.992965] Key type ._llcrypt unregistered [12962.131663] Key type ._llcrypt registered [12962.135709] Key type .llcrypt registered [12962.225184] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [12973.625691] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [12974.297310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12974.358581] alg: No test for adler32 (adler32-zlib) [12975.307574] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [12975.455230] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [12977.111327] Key type lgssc registered [12977.949973] Lustre: Echo OBD driver; http://www.lustre.org/ [13001.091493] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13001.137571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13002.307332] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13002.334284] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13002.396582] Lustre: tst250fs-MDT0000: new disk, initializing [13002.562449] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13002.574122] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13005.257215] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [13012.219397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13012.296862] Lustre: 389686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13012.329994] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13012.333658] Lustre: Skipped 1 previous similar message [13012.376331] Lustre: tst250fs-MDT0001: new disk, initializing [13012.429766] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13012.474653] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13012.487223] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13015.413879] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [13018.856846] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13023.294350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13023.493358] Lustre: tst250fs-OST0000: new disk, initializing [13023.495558] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13023.500778] Lustre: 391315:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13023.587344] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13027.537634] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [13028.914196] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13028.924361] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13029.002418] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13035.117258] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13035.282277] Lustre: tst250fs-OST0001: new disk, initializing [13035.286777] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13035.295818] Lustre: 392185:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13035.363302] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13037.842543] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13037.848327] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13037.910316] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13039.478640] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [13043.638703] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13063.648642] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13063.652885] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13063.659454] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13063.659940] Lustre: Skipped 3 previous similar messages [13063.676410] Lustre: Skipped 3 previous similar messages [13068.770716] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13068.775307] Lustre: Skipped 3 previous similar messages [13073.378598] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13073.497241] Lustre: server umount tst250fs-MDT0000 complete [13076.069863] LustreError: 389677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790992296 with bad export cookie 1412592800862526761 [13076.071111] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13076.077888] LustreError: 389677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13076.326591] Lustre: server umount tst250fs-MDT0001 complete [13089.500440] Lustre: server umount tst250fs-OST0000 complete [13102.363513] Lustre: server umount tst250fs-OST0001 complete [13114.489964] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [13120.503979] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [13142.734780] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 21:52:41 (1790992361) [13143.885871] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13155.957505] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [13163.763960] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [13185.682850] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12886 sec ========= 21:53:24 (1790992404) [13186.821535] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:53:26 (1790992406) === [13189.279854] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:53:28 (1790992408) === [13216.112931] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [13217.696560] Key type lgssc unregistered [13217.925357] LNet: 401332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13217.928923] LNetError: 401332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13217.938551] LNet: Removed LNI 192.168.202.160@tcp [13218.531588] Key type .llcrypt unregistered [13218.533951] Key type ._llcrypt unregistered