[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 486775736 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002252] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004018] kvm-guest: setup PV IPIs [ 0.007762] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.011141] LSM: Security Framework initializing [ 0.012058] Yama: becoming mindful. [ 0.014036] SELinux: Initializing. [ 0.015123] *** VALIDATE selinux *** [ 0.024541] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028720] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029144] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030141] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031146] *** VALIDATE tmpfs *** [ 0.033537] *** VALIDATE proc *** [ 0.035097] *** VALIDATE cgroup *** [ 0.036012] *** VALIDATE cgroup2 *** [ 0.037259] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038114] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040024] Spectre V2 : User space: Vulnerable [ 0.041006] Speculative Store Bypass: Vulnerable [ 0.044263] debug: unmapping init [mem 0xffffffffb6a59000-0xffffffffb6a60fff] [ 0.046877] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047550] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048019] ... version: 2 [ 0.048896] ... bit width: 48 [ 0.049017] ... generic registers: 4 [ 0.050016] ... value mask: 0000ffffffffffff [ 0.051009] ... max period: 00007fffffffffff [ 0.052017] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054325] rcu: Hierarchical SRCU implementation. [ 0.056289] smp: Bringing up secondary CPUs ... [ 0.057553] x86: Booting SMP configuration: [ 0.058037] .... node #0, CPUs: #1 #2 #3 [ 0.061468] smp: Brought up 1 node, 4 CPUs [ 0.062971] smpboot: Max logical packages: 1 [ 0.063014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141342] node 0 deferred pages initialised in 75ms [ 0.145211] devtmpfs: initialized [ 0.147305] x86/mm: Memory block size: 128MB [ 0.151000] gcov: version magic: 0x41383552 [ 0.154279] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.160120] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.163445] pinctrl core: initialized pinctrl subsystem [ 0.166237] [ 0.167008] ************************************************************* [ 0.171015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174013] ** ** [ 0.178014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.181016] ** ** [ 0.185017] ** This means that this kernel is built to expose internal ** [ 0.188015] ** IOMMU data structures, which may compromise security on ** [ 0.191021] ** your system. ** [ 0.195016] ** ** [ 0.198018] ** If you see this message and you are not debugging the ** [ 0.202018] ** kernel, report this immediately to your vendor! ** [ 0.204016] ** ** [ 0.207017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.211032] ************************************************************* [ 0.214794] NET: Registered protocol family 16 [ 0.217584] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.221062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.225073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.229754] cpuidle: using governor menu [ 0.232984] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.236506] PCI: Using configuration type 1 for base access [ 0.240205] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.250147] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.253080] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.256195] cryptd: max_cpu_qlen set to 1000 [ 0.260186] ACPI: Added _OSI(Module Device) [ 0.263033] ACPI: Added _OSI(Processor Device) [ 0.264037] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.266038] ACPI: Added _OSI(Processor Aggregator Device) [ 0.271806] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.280604] ACPI: Interpreter enabled [ 0.282107] ACPI: PM: (supports S0 S3 S4 S5) [ 0.284020] ACPI: Using IOAPIC for interrupt routing [ 0.286205] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.289532] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.300670] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.303066] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.306032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.310115] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.315156] acpiphp: Slot [2] registered [ 0.317136] acpiphp: Slot [5] registered [ 0.318191] acpiphp: Slot [6] registered [ 0.320108] acpiphp: Slot [7] registered [ 0.321114] acpiphp: Slot [8] registered [ 0.323150] acpiphp: Slot [9] registered [ 0.325141] acpiphp: Slot [10] registered [ 0.326124] acpiphp: Slot [3] registered [ 0.328257] acpiphp: Slot [4] registered [ 0.329110] acpiphp: Slot [11] registered [ 0.330263] acpiphp: Slot [12] registered [ 0.332104] acpiphp: Slot [13] registered [ 0.333082] acpiphp: Slot [14] registered [ 0.335123] acpiphp: Slot [15] registered [ 0.337120] acpiphp: Slot [16] registered [ 0.339145] acpiphp: Slot [17] registered [ 0.340138] acpiphp: Slot [18] registered [ 0.342209] acpiphp: Slot [19] registered [ 0.344200] acpiphp: Slot [20] registered [ 0.346182] acpiphp: Slot [21] registered [ 0.347128] acpiphp: Slot [22] registered [ 0.349164] acpiphp: Slot [23] registered [ 0.351134] acpiphp: Slot [24] registered [ 0.353147] acpiphp: Slot [25] registered [ 0.354388] acpiphp: Slot [26] registered [ 0.356132] acpiphp: Slot [27] registered [ 0.358202] acpiphp: Slot [28] registered [ 0.360140] acpiphp: Slot [29] registered [ 0.361154] acpiphp: Slot [30] registered [ 0.363231] acpiphp: Slot [31] registered [ 0.365255] PCI host bridge to bus 0000:00 [ 0.367042] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.370078] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.373046] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.377041] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.379033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.382030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.384237] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.387234] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.390171] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.401023] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.404864] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.407025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.410029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.412022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.415768] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.419022] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.421038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.425633] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.428937] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.439752] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.443012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.447128] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.457021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.467017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.482025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.490621] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.504029] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.511024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.534021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.543000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.553019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.566021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.594018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.605187] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.619033] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.642033] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.657026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.667217] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.676023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.682034] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.703020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.711290] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.720029] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.724018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.734021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.744484] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.746393] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.749533] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.752424] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.754168] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.758248] iommu: Default domain type: Passthrough [ 0.759537] SCSI subsystem initialized [ 0.761251] ACPI: bus type USB registered [ 0.763168] usbcore: registered new interface driver usbfs [ 0.765133] usbcore: registered new interface driver hub [ 0.767122] usbcore: registered new device driver usb [ 0.769222] pps_core: LinuxPPS API ver. 1 registered [ 0.770012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.772102] PTP clock support registered [ 0.774047] EDAC MC: Ver: 3.0.0 [ 0.776125] PCI: Using ACPI for IRQ routing [ 0.778939] NetLabel: Initializing [ 0.780014] NetLabel: domain hash size = 128 [ 0.781012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.783104] NetLabel: unlabeled traffic allowed by default [ 0.786167] vgaarb: loaded [ 0.787293] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.789017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.797136] clocksource: Switched to clocksource kvm-clock [ 0.908688] VFS: Disk quotas dquot_6.6.0 [ 0.910387] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.912926] *** VALIDATE ramfs *** [ 0.913841] *** VALIDATE hugetlbfs *** [ 0.915306] pnp: PnP ACPI init [ 0.917445] pnp: PnP ACPI: found 6 devices [ 0.935130] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.938580] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.940865] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.943126] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.945639] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.948082] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.951047] NET: Registered protocol family 2 [ 0.953507] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.958677] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.962364] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.967590] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.971156] TCP: Hash tables configured (established 65536 bind 65536) [ 0.974037] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.977450] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.980343] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.983441] NET: Registered protocol family 1 [ 0.985856] RPC: Registered named UNIX socket transport module. [ 0.988129] RPC: Registered udp transport module. [ 0.989996] RPC: Registered tcp transport module. [ 0.991824] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.994132] NET: Registered protocol family 44 [ 0.995820] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.997740] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.999922] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.002225] PCI: CLS 0 bytes, default 64 [ 1.004507] Unpacking initramfs... [ 2.502837] debug: unmapping init [mem 0xffff9228bcc54000-0xffff9228bffbffff] [ 2.509486] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.511399] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.515730] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.001766] Initialise system trusted keyrings [ 3.003426] Key type blacklist registered [ 3.005344] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.018348] zbud: loaded [ 3.022206] *** VALIDATE nfs *** [ 3.023109] *** VALIDATE nfs4 *** [ 3.025023] pstore: using deflate compression [ 3.028067] Platform Keyring initialized [ 3.140139] NET: Registered protocol family 38 [ 3.141615] Key type asymmetric registered [ 3.143013] Asymmetric key parser 'x509' registered [ 3.144722] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.147468] io scheduler mq-deadline registered [ 3.148863] io scheduler kyber registered [ 3.150243] io scheduler bfq registered [ 3.151991] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.154837] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.157627] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.160378] ACPI: Power Button [PWRF] [ 3.166289] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.172950] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.191616] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.201259] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.221038] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.247291] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.274324] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.279803] Non-volatile memory driver v1.3 [ 3.281505] Linux agpgart interface v0.103 [ 3.329181] virtio_blk virtio1: [vda] 150688 512-byte logical blocks (77.2 MB/73.6 MiB) [ 3.332112] vda: detected capacity change from 0 to 77152256 [ 3.350592] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.355344] vdb: detected capacity change from 0 to 1073741824 [ 3.371861] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.374502] vdc: detected capacity change from 0 to 2621440000 [ 3.389648] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.392618] vdd: detected capacity change from 0 to 2621440000 [ 3.407378] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.409181] vde: detected capacity change from 0 to 4294967296 [ 3.423100] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.425064] vdf: detected capacity change from 0 to 4294967296 [ 3.432825] libphy: Fixed MDIO Bus: probed [ 3.439489] usbcore: registered new interface driver usbserial_generic [ 3.441551] usbserial: USB Serial support registered for generic [ 3.443903] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.448553] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.450268] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.452687] mousedev: PS/2 mouse device common for all mice [ 3.455808] rtc_cmos 00:05: RTC can wake from S4 [ 3.458756] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.464829] rtc_cmos 00:05: registered as rtc0 [ 3.465723] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.469760] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.472609] intel_pstate: CPU model not supported [ 3.477899] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.489956] hid: raw HID events driver (C) Jiri Kosina [ 3.496085] usbcore: registered new interface driver usbhid [ 3.498544] usbhid: USB HID core driver [ 3.500423] drop_monitor: Initializing network drop monitor service [ 3.502959] Initializing XFRM netlink socket [ 3.505129] NET: Registered protocol family 10 [ 3.508539] Segment Routing with IPv6 [ 3.510037] NET: Registered protocol family 17 [ 3.513865] mpls_gso: MPLS GSO support [ 3.521615] RAS: Correctable Errors collector initialized. [ 3.523642] AVX version of gcm_enc/dec engaged. [ 3.525558] AES CTR mode by8 optimization enabled [ 3.620984] sched_clock: Marking stable (3620952273, 0)->(4433018050, -812065777) [ 3.625950] registered taskstats version 1 [ 3.629436] Loading compiled-in X.509 certificates [ 3.632344] zswap: loaded using pool lzo/zbud [ 3.664547] Key type big_key registered [ 3.677205] Key type encrypted registered [ 3.678836] ima: No TPM chip found, activating TPM-bypass! [ 3.680840] ima: Allocated hash algorithm: sha1 [ 3.682518] ima: No architecture policies found [ 3.684482] evm: Initialising EVM extended attributes: [ 3.686547] evm: security.selinux [ 3.687851] evm: security.ima [ 3.689008] evm: security.capability [ 3.690401] evm: HMAC attrs: 0x1 [ 3.694279] rtc_cmos 00:05: setting system clock to 2026-09-30 12:54:52 UTC (1790772892) [ 3.700556] debug: unmapping init [mem 0xffffffffb7a03000-0xffffffffb7bfffff] [ 3.703750] debug: unmapping init [mem 0xffffffffb6782000-0xffffffffb6a58fff] [ 3.715333] Write protecting the kernel read-only data: 28672k [ 3.718968] debug: unmapping init [mem 0xffffffffb4e03000-0xffffffffb4ffffff] [ 3.722147] debug: unmapping init [mem 0xffffffffb5714000-0xffffffffb57fffff] [ 3.762206] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.770522] systemd[1]: Detected virtualization kvm. [ 3.773161] systemd[1]: Detected architecture x86-64. [ 3.775042] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.803393] systemd[1]: No hostname configured. [ 3.804680] systemd[1]: Set hostname to . [ 3.806461] random: systemd: uninitialized urandom read (16 bytes read) [ 3.808314] systemd[1]: Initializing machine ID from random generator. [ 3.892946] random: ln: uninitialized urandom read (6 bytes read) [ 3.995666] random: systemd: uninitialized urandom read (16 bytes read) [ 3.997704] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.005599] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 4.010899] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.633064] device-mapper: uevent: version 1.0.3 [ 4.635250] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.250851] virtio_net virtio0 ens2: renamed from eth0 [ 5.263603] random: fast init done [ 5.343532] scsi host0: ata_piix [ 5.380841] scsi host1: ata_piix [ 5.382680] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.385348] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.281635] dracut-initqueue[579]: RTNETLINK answers: File exists [ 10.197447] random: crng init done [ 10.198854] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.314621] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.639186] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.905758] SELinux: Disabled at runtime. [ 12.965956] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.977000] systemd[1]: Detected virtualization kvm. [ 12.979565] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.547957] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.552144] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.557213] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.563348] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.567137] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.576344] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.599572] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapp[ 13.779471] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS er. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 14.175053] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.563881] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.609450] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.724294] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.742200] EDAC sbridge: Ver: 1.1.2 [ 16.339229] Key type dns_resolver registered [ 16.654164] NFS: Registering the id_resolver key type [ 16.656289] Key type id_resolver registered [ 16.657960] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg104-server login: [ 43.256321] libcfs: loading out-of-tree module taints kernel. [ 43.278092] Key type ._llcrypt registered [ 43.279905] Key type .llcrypt registered [ 43.333681] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 62.755944] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 64.375585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 64.388247] alg: No test for adler32 (adler32-zlib) [ 65.857067] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 66.901080] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 68.688403] Key type lgssc registered [ 69.323927] hrtimer: interrupt took 3905479 ns [ 70.961379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 91.445257] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 145.293276] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 162.404689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 162.442152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 163.674768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 163.711572] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 163.798218] Lustre: lustre-MDT0000: new disk, initializing [ 163.872069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 163.884823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 169.383611] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.200696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 185.374301] Lustre: 6508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 185.401897] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 185.405447] Lustre: Skipped 1 previous similar message [ 185.534619] Lustre: lustre-MDT0001: new disk, initializing [ 185.634966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 185.697973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 185.709704] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 191.010840] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.952501] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 208.356058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 208.738787] Lustre: lustre-OST0000: new disk, initializing [ 208.742981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 208.752384] Lustre: 8445:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 208.828882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 215.626759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 215.648972] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 215.801424] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 216.658052] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.340587] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 233.533556] Lustre: lustre-OST0001: new disk, initializing [ 233.553791] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 233.559569] Lustre: 9518:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 233.660494] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 240.260110] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.712781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 240.724565] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 240.793914] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 254.361714] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 265.017791] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 271.441984] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing check_logdir /tmp/testlogs/ [ 276.888989] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing yml_node [ 282.472964] Lustre: DEBUG MARKER: Client: 2.17.59.4 [ 287.260964] Lustre: DEBUG MARKER: MDS: 2.17.59.4 [ 290.389917] Lustre: DEBUG MARKER: OSS: 2.17.59.4 [ 292.474090] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 30 08:59:38 EDT 2026 [ 311.737917] Lustre: DEBUG MARKER: only running tests: 121-999 [ 313.282523] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 315.010492] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 333.284366] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.290659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 333.301609] Lustre: Skipped 3 previous similar messages [ 333.321193] Lustre: Skipped 3 previous similar messages [ 337.577724] Lustre: server umount lustre-MDT0000 complete [ 341.037223] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790773229 with bad export cookie 11188924238075669689 [ 341.042131] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 341.054156] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 343.526554] LustreError: 10095:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.527241] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.536670] LustreError: 10095:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 343.554127] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 346.529184] LustreError: 6513:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 346.533235] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 346.560981] LustreError: 6513:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 346.573867] Lustre: Skipped 2 previous similar messages [ 347.571905] Lustre: server umount lustre-MDT0001 complete [ 352.583711] Lustre: server umount lustre-OST0000 complete [ 357.271866] Lustre: server umount lustre-OST0001 complete [ 366.035631] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 377.402514] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 418.409862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 418.648221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 418.686174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 418.777939] Lustre: lustre-MDT0000: new disk, initializing [ 418.884667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 418.918549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 424.232140] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 437.308287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 437.506790] Lustre: 15137:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 437.601904] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 437.605387] Lustre: Skipped 1 previous similar message [ 437.714370] Lustre: lustre-MDT0001: new disk, initializing [ 437.850581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 437.913902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 437.938934] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 443.427308] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 453.697558] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 460.587814] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 468.834084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 469.173881] Lustre: lustre-OST0000: new disk, initializing [ 469.182505] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 469.190700] Lustre: 17105:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 469.320336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 470.385804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 470.409164] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 470.578239] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 477.181763] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 490.098570] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 495.762474] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 496.027790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 501.311886] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 501.520640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 506.340494] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.343699] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 506.354550] Lustre: Skipped 1 previous similar message [ 506.356195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 506.370265] LustreError: Skipped 1 previous similar message [ 506.390888] Lustre: Skipped 1 previous similar message [ 509.948765] Lustre: server umount lustre-OST0000 complete [ 516.579619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.609190] Lustre: Skipped 1 previous similar message [ 516.611822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 516.642506] Lustre: Skipped 1 previous similar message [ 521.698455] LustreError: 17405:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.718693] LustreError: 17405:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 521.863869] Lustre: server umount lustre-MDT0000 complete [ 526.130392] LustreError: 15130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790773414 with bad export cookie 11188924238075671222 [ 526.139386] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 526.140095] LustreError: 15130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 526.489264] Lustre: server umount lustre-MDT0001 complete [ 532.990266] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 09:03:39 (1790773419) [ 554.699669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.492972] LustreError: 19982:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 555.642360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 560.670656] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 561.124692] LustreError: 19983:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.343041] LustreError: 19983:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.360660] LustreError: 19983:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 571.227373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 576.689992] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 587.224340] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 594.374745] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 597.053714] Lustre: Failing over lustre-MDT0000 [ 597.384200] Lustre: server umount lustre-MDT0000 complete [ 597.480799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 597.491618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 597.524997] LustreError: 20711:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 617.956645] Lustre: 3636:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790773490/real 1790773490] req@ffff9228037e6d80 x1877761542234112/t0(0) o400->MGC192.168.201.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790773506 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 617.998819] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 618.830975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 628.210398] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x9b470b502542383c [ 628.240718] Lustre: MGC192.168.201.104@tcp: Connection restored to 0@lo (at 0@lo) [ 628.640655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 628.645821] Lustre: Skipped 1 previous similar message [ 628.734769] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 633.826104] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 633.847470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 633.880755] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 634.334960] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 646.032653] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 652.260116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 652.269987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 652.284160] Lustre: Skipped 2 previous similar messages [ 657.014761] Lustre: server umount lustre-MDT0000 complete [ 657.377675] LustreError: 19983:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 657.387302] LustreError: 19983:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 662.218359] LustreError: 19962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790773551 with bad export cookie 11188924238075672636 [ 662.228653] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 662.234923] LustreError: 19962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 662.805182] Lustre: server umount lustre-MDT0001 complete [ 673.630824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 679.599297] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 689.524161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 695.265253] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 703.971597] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 710.965491] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 713.953658] Lustre: Failing over lustre-MDT0000 [ 714.356577] Lustre: server umount lustre-MDT0000 complete [ 714.727063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 725.986720] LustreError: 24057:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 726.014951] LustreError: 24057:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 731.104837] Lustre: 3635:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790773603/real 1790773603] req@ffff922941d71f80 x1877761542265728/t0(0) o400->MGC192.168.201.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790773619 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 731.144177] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 734.074557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 740.388922] LustreError: 3634:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9228037e5180 x1877761542268800/t0(0) o250->MGC192.168.201.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 740.808635] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 745.494380] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 745.963403] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 745.980046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 745.984562] Lustre: Skipped 1 previous similar message [ 745.998465] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 755.321558] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 760.419530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 760.442894] Lustre: Skipped 1 previous similar message [ 760.451801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 760.459624] Lustre: Skipped 1 previous similar message [ 765.642730] Lustre: server umount lustre-MDT0000 complete [ 769.715211] LustreError: 24065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790773658 with bad export cookie 11188924238075673581 [ 769.720745] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 769.727291] LustreError: 24065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 770.122523] Lustre: server umount lustre-MDT0001 complete [ 779.975214] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 09:07:46 (1790773666) [ 798.233333] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 810.549768] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 857.187267] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 869.134694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 869.423474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 869.471330] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 869.551323] Lustre: lustre-MDT0000: new disk, initializing [ 869.660697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 869.670623] Lustre: Skipped 3 previous similar messages [ 869.685450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 875.136569] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 889.144776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 889.361298] Lustre: 29704:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 889.719649] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 895.092523] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 905.388895] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 912.623137] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 920.389097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 920.722322] Lustre: 31673:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 922.332370] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 922.385477] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 922.470512] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 922.495048] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 926.790499] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 936.578561] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 942.445294] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 963.558790] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 963.561499] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 963.571055] Lustre: Skipped 1 previous similar message [ 963.574944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 963.579059] LustreError: Skipped 1 previous similar message [ 963.602557] Lustre: Skipped 1 previous similar message [ 967.684756] Lustre: server umount lustre-OST0000 complete [ 971.745677] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 978.180456] Lustre: server umount lustre-MDT0000 complete [ 978.913943] LustreError: 30575:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 978.920670] LustreError: 30575:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 983.777388] LustreError: 29696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790773872 with bad export cookie 11188924238075673987 [ 983.809825] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 983.819535] LustreError: 29696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 984.844672] Lustre: server umount lustre-MDT0001 complete [ 998.277602] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 1001.528314] Key type lgssc unregistered [ 1001.770170] LNet: 33654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1001.777724] LNetError: 33654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1001.786762] LNet: Removed LNI 192.168.201.104@tcp [ 1002.329169] Key type .llcrypt unregistered [ 1002.330626] Key type ._llcrypt unregistered [ 1030.746977] Key type ._llcrypt registered [ 1030.749244] Key type .llcrypt registered [ 1030.841739] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 1032.465966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1032.502362] alg: No test for adler32 (adler32-zlib) [ 1033.548361] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 1033.783382] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 1035.472215] Key type lgssc registered [ 1036.912064] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1045.151312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1045.185580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1046.788915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1051.519946] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1056.227357] LustreError: 35083:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1056.246435] LustreError: 35083:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1059.973335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1060.446213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1064.949519] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1074.532696] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1080.202204] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1088.800165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1089.505646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1096.738372] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1105.977824] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1106.118683] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1121.836100] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:13:27 (1790774007) [ 1124.439403] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1126.200417] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1133.271924] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:13:40 (1790774020) [ 1146.379618] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:13:53 (1790774033) [ 1148.730190] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1160.575454] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1160.599420] Lustre: Skipped 2 previous similar messages [ 1162.444853] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 09:14:09 (1790774049) [ 1166.412421] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1166.418303] Lustre: Skipped 3 previous similar messages [ 1185.297702] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1185.301945] Lustre: Skipped 6 previous similar messages [ 1188.413969] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 09:14:35 (1790774075) [ 1192.494404] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1196.462122] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1205.067223] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 09:14:52 (1790774092) [ 1207.246394] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1207.253369] Lustre: Skipped 3 previous similar messages [ 1221.927172] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:15:08 (1790774108) [ 1240.036781] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1240.039240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1240.055602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1244.132714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1244.134518] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1244.154516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1245.505986] Lustre: server umount lustre-OST0000 complete [ 1249.250694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1249.258754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1249.261364] Lustre: Skipped 1 previous similar message [ 1249.271929] Lustre: Skipped 2 previous similar messages [ 1254.375232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1255.374133] Lustre: server umount lustre-MDT0000 complete [ 1259.495031] LustreError: 35084:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1259.512415] LustreError: 35084:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1259.753061] LustreError: 35065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790774148 with bad export cookie 5438870998774782694 [ 1259.756859] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1259.760035] LustreError: 35065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1260.401145] Lustre: server umount lustre-MDT0001 complete [ 1273.280783] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 1276.158793] Key type lgssc unregistered [ 1276.547361] LNet: 42942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1276.559395] LNetError: 42942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1277.607979] LNet: Removed LNI 192.168.201.104@tcp [ 1278.578850] Key type .llcrypt unregistered [ 1278.580727] Key type ._llcrypt unregistered [ 1298.765903] Key type ._llcrypt registered [ 1298.767156] Key type .llcrypt registered [ 1298.860733] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 1317.100547] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 1318.937043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1318.970722] alg: No test for adler32 (adler32-zlib) [ 1319.983479] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 1320.194144] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 1321.873111] Key type lgssc registered [ 1323.598726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1364.349908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1364.371556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1365.693280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1365.730281] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1365.861055] Lustre: lustre-MDT0000: new disk, initializing [ 1365.987408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1366.014739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1372.042788] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1383.359259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1383.545712] Lustre: 46113:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1383.614132] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1383.628929] Lustre: Skipped 1 previous similar message [ 1383.756992] Lustre: lustre-MDT0001: new disk, initializing [ 1383.925507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1383.970894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1384.001348] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1389.399055] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1399.148773] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1404.840449] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1411.484988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1411.767491] Lustre: lustre-OST0000: new disk, initializing [ 1411.771846] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1411.781303] Lustre: 48082:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1411.903756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1418.841528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1418.861553] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1418.944970] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1419.004811] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1429.042631] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1433.312898] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1433.548911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1438.734246] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1438.955725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1444.834368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1444.838287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1444.852541] Lustre: Skipped 1 previous similar message [ 1444.854656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1444.887806] Lustre: Skipped 1 previous similar message [ 1447.214870] Lustre: server umount lustre-OST0000 complete [ 1455.074886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1455.076924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1455.093843] Lustre: Skipped 1 previous similar message [ 1460.194891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1460.202330] Lustre: Skipped 2 previous similar messages [ 1464.291652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1464.303228] Lustre: Skipped 1 previous similar message [ 1465.312617] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1465.571907] Lustre: server umount lustre-MDT0000 complete [ 1469.239924] LustreError: 46105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790774358 with bad export cookie 4108007278231312490 [ 1469.241446] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1469.257458] LustreError: 46105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1469.690828] Lustre: server umount lustre-MDT0001 complete [ 1480.942548] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:19:27 (1790774367) [ 1490.707334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1491.252557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1496.564658] LustreError: 50542:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1496.638280] LustreError: 50542:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1498.123738] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1501.667088] LustreError: 50543:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1506.785733] LustreError: 50542:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1509.530145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1510.104198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1516.183958] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1526.353969] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1531.980766] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1539.687014] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1540.232332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1547.689979] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1556.952811] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1566.087841] LustreError: 53074:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1567.005634] Lustre: Modifying parameter general.timeout=1 in log params [ 1569.220896] Lustre: Modifying parameter general.timeout=3 in log params [ 1569.226808] Lustre: Skipped 1 previous similar message [ 1572.079763] Lustre: Modifying parameter general.timeout=6 in log params [ 1572.084848] Lustre: Skipped 2 previous similar messages [ 1576.771113] Lustre: Modifying parameter general.timeout=11 in log params [ 1576.775782] Lustre: Skipped 4 previous similar messages [ 1585.494977] Lustre: Modifying parameter general.timeout=20 in log params [ 1585.498946] Lustre: Skipped 8 previous similar messages [ 1602.295803] Lustre: Modifying parameter general.timeout=38 in log params [ 1602.301810] Lustre: Skipped 17 previous similar messages [ 1634.697419] Lustre: Modifying parameter general.timeout=71 in log params [ 1634.703199] Lustre: Skipped 32 previous similar messages [ 1704.045692] Lustre: Disabling parameter general.timeout= in log params [ 1704.048666] Lustre: Skipped 58 previous similar messages [ 1707.655272] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:23:14 (1790774594) [ 1711.074111] Lustre: 60034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1713.032553] Lustre: 60133:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1717.362527] Lustre: 60329:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1717.382850] Lustre: 60329:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1721.979618] Lustre: 60542:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1721.989930] Lustre: 60542:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1751.400768] Lustre: 61911:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1751.435209] LustreError: 61911:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1755.117207] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:24:02 (1790774642) [ 1768.429034] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1768.431410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1768.452658] Lustre: Skipped 1 previous similar message [ 1773.797578] Lustre: server umount lustre-MDT0000 complete [ 1778.089868] LustreError: 50523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790774666 with bad export cookie 4108007278231313372 [ 1778.092543] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1778.102967] LustreError: 50523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1778.409237] Lustre: server umount lustre-MDT0001 complete [ 1793.717687] Lustre: server umount lustre-OST0000 complete [ 1826.711085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1827.119987] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1827.492249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1833.112431] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1843.620025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1843.695586] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1843.732495] Lustre: 64441:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1849.251973] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1856.883593] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1863.229232] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1870.579826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1870.739754] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1871.080758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1871.096511] Lustre: Skipped 1 previous similar message [ 1877.928735] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1885.852749] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1887.673855] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1887.687519] Lustre: Skipped 23 previous similar messages [ 1887.805953] Lustre: 67007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1898.465898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1898.474532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1898.488884] Lustre: Skipped 1 previous similar message [ 1898.503226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1898.517259] Lustre: Skipped 5 previous similar messages [ 1901.132455] Lustre: server umount lustre-OST0000 complete [ 1905.632979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1905.639776] LustreError: Skipped 1 previous similar message [ 1905.648924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1905.673037] Lustre: Skipped 1 previous similar message [ 1911.126069] Lustre: server umount lustre-MDT0000 complete [ 1914.343766] LustreError: 64451:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1914.374666] LustreError: 64451:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1915.021703] LustreError: 65190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790774803 with bad export cookie 4108007278231316529 [ 1915.031208] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1915.033260] LustreError: 65190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1915.348687] Lustre: server umount lustre-MDT0001 complete [ 1924.908684] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 1927.118058] Key type lgssc unregistered [ 1927.426269] LNet: 68212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1927.433168] LNetError: 68212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1927.450991] LNet: Removed LNI 192.168.201.104@tcp [ 1928.541508] Key type .llcrypt unregistered [ 1928.553090] Key type ._llcrypt unregistered [ 1960.448817] Key type ._llcrypt registered [ 1960.452513] Key type .llcrypt registered [ 1960.528410] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 1961.694478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1961.741595] alg: No test for adler32 (adler32-zlib) [ 1962.813609] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 1963.202488] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 1964.976495] Key type lgssc registered [ 1966.899778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1976.946154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1976.980646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1978.741198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1983.331977] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1989.097970] LustreError: 69642:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1989.130371] LustreError: 69642:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1992.351857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1993.005274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1998.089125] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2006.682146] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2012.599064] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2019.849560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2020.446331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2027.732671] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2035.766213] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2043.618951] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2053.616836] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2053.624310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2053.629428] Lustre: Skipped 2 previous similar messages [ 2053.642290] Lustre: Skipped 2 previous similar messages [ 2058.082319] Lustre: server umount lustre-MDT0000 complete [ 2058.723785] LustreError: 69639:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2058.745829] LustreError: 69639:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2061.504883] LustreError: 71497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790774950 with bad export cookie 2074001046767664411 [ 2061.510872] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2061.522097] LustreError: 71497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2061.829489] Lustre: server umount lustre-MDT0001 complete [ 2075.696248] Lustre: server umount lustre-OST0000 complete [ 2104.103339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2104.415985] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2104.459783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2104.748794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2110.600350] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2119.780842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2119.856855] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2119.881059] Lustre: 74514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2120.290433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2125.348885] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2133.568894] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2140.174133] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2147.921451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2148.176702] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2148.653550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2155.238749] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2164.092923] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2165.723695] Lustre: 77090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2168.131814] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2175.971146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2175.981373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2175.993637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2176.496164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2181.292986] Lustre: server umount lustre-OST0000 complete [ 2186.721618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2186.722231] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2186.742094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2186.754843] Lustre: Skipped 2 previous similar messages [ 2186.759399] Lustre: Skipped 2 previous similar messages [ 2191.850896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2191.868924] Lustre: Skipped 2 previous similar messages [ 2192.291898] Lustre: server umount lustre-MDT0000 complete [ 2196.612667] LustreError: 74508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790775085 with bad export cookie 2074001046767665258 [ 2196.623507] LustreError: 74508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2196.632438] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2196.978070] Lustre: server umount lustre-MDT0001 complete [ 2211.127913] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 2214.672059] Key type lgssc unregistered [ 2214.911217] LNet: 78302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2214.923697] LNetError: 78302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2214.943885] LNet: Removed LNI 192.168.201.104@tcp [ 2216.034973] Key type .llcrypt unregistered [ 2216.037518] Key type ._llcrypt unregistered [ 2243.617055] Key type ._llcrypt registered [ 2243.619153] Key type .llcrypt registered [ 2243.814417] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 2246.031691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2246.088435] alg: No test for adler32 (adler32-zlib) [ 2247.252394] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 2247.690513] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 2249.408280] Key type lgssc registered [ 2251.364646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2260.134819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2260.171788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2261.928124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2266.618676] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2271.206646] LustreError: 79740:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2271.222380] LustreError: 79740:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2276.326439] LustreError: 79739:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2276.743459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2277.905306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2283.206675] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2292.308552] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2297.991974] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2306.263333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2306.533060] LustreError: 81569:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2306.919450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2313.848625] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2326.020583] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2333.676360] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2334.710784] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2336.489180] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2336.499178] Lustre: Skipped 1 previous similar message [ 2339.226442] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2339.236237] Lustre: Skipped 1 previous similar message [ 2343.489499] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2343.498895] Lustre: Skipped 3 previous similar messages [ 2351.725991] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2351.730807] Lustre: Skipped 7 previous similar messages [ 2368.812963] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2368.823119] Lustre: Skipped 15 previous similar messages [ 2401.452921] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2401.455024] Lustre: Skipped 28 previous similar messages [ 2448.756346] Lustre: 87852:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2451.769410] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 09:35:38 (1790775338) [ 2453.581834] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2456.020670] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 09:35:42 (1790775342) [ 2460.643969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2460.651772] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2460.656088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2460.657029] Lustre: Skipped 1 previous similar message [ 2460.682050] LustreError: Skipped 1 previous similar message [ 2465.764957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2465.780479] Lustre: Skipped 2 previous similar messages [ 2466.720552] Lustre: server umount lustre-OST0000 complete [ 2471.923108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2471.933632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.941711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2476.004468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2476.005194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2477.447814] Lustre: server umount lustre-MDT0000 complete [ 2481.121793] LustreError: 80472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2481.148201] LustreError: 80472:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2481.632569] LustreError: 79722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790775370 with bad export cookie 5256496240318361663 [ 2481.638759] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2481.641937] LustreError: 79722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2482.052775] Lustre: server umount lustre-MDT0001 complete [ 2494.314479] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 2497.360486] Key type lgssc unregistered [ 2497.646085] LNet: 89101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2497.657110] LNetError: 89101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2497.667081] LNet: Removed LNI 192.168.201.104@tcp [ 2498.468808] Key type .llcrypt unregistered [ 2498.470596] Key type ._llcrypt unregistered [ 2503.920291] Key type ._llcrypt registered [ 2503.922492] Key type .llcrypt registered [ 2504.012475] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2509.300679] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules [ 2511.775513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2511.814238] alg: No test for adler32 (adler32-zlib) [ 2513.130211] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 2513.551721] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 2515.400968] Key type lgssc registered [ 2518.044568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2520.443699] LustreError: 90373:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2542.449825] LustreError: 90373:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2543.177481] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 2553.419939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2553.458921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2555.181962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2560.266516] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2564.588626] LustreError: 91395:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2564.629231] LustreError: 91395:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2568.703360] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 09:37:35 (1790775455) [ 2569.699247] LustreError: 91396:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2574.825680] LustreError: 91395:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2575.041955] LustreError: 91427:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2575.058552] LustreError: 91427:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2575.065059] LustreError: 91427:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2575.458811] Lustre: server umount lustre-MDT0000 complete [ 2592.243737] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 2595.380713] Key type lgssc unregistered [ 2595.736981] LNet: 92920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2595.754209] LNetError: 92920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2595.770166] LNet: Removed LNI 192.168.201.104@tcp [ 2596.893275] Key type .llcrypt unregistered [ 2596.895422] Key type ._llcrypt unregistered [ 2618.499873] Key type ._llcrypt registered [ 2618.502223] Key type .llcrypt registered [ 2618.655797] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 2619.568956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2619.618800] alg: No test for adler32 (adler32-zlib) [ 2620.539189] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 2620.719381] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 2622.344313] Key type lgssc registered [ 2623.827579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2632.956965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2632.986959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2634.665934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2640.623260] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2644.961545] LustreError: 94154:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2644.991569] LustreError: 94154:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2650.082632] LustreError: 94155:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2653.822156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2654.873193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2659.737615] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2667.839240] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2674.769351] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2682.134182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2682.896199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2690.359441] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2698.224928] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2754.062643] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 09:40:40 (1790775640) [ 2756.243051] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2758.321901] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 09:40:44 (1790775644) [ 2762.209708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2762.225682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.239027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2765.799587] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2765.801152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2765.821476] Lustre: Skipped 1 previous similar message [ 2765.850855] Lustre: Skipped 2 previous similar messages [ 2768.065077] Lustre: server umount lustre-MDT0000 complete [ 2770.914397] LustreError: 94150:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2770.941989] LustreError: 94150:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2771.709954] LustreError: 94135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790775660 with bad export cookie 7678838592011984321 [ 2771.713252] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2771.719195] LustreError: 94135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2772.031616] Lustre: server umount lustre-MDT0001 complete [ 2786.177618] Lustre: server umount lustre-OST0000 complete [ 2797.089605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2797.683413] LustreError: 98472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2797.913496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2803.176979] LustreError: 98473:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2804.037398] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2812.392166] LustreError: 98472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2812.425907] LustreError: 98472:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2815.785219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2816.281292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2820.253911] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2828.231715] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2835.254420] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2849.037811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2849.258622] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2849.263333] LustreError: 98461:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2849.276341] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2849.282352] LustreError: 100557:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2849.286834] LustreError: 100557:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2849.291189] LustreError: 100557:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2849.343533] Lustre: server umount lustre-OST0000 complete [ 2849.347371] LustreError: 100557:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2858.392655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2858.611113] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2858.631358] LustreError: 98460:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2858.700471] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2858.715953] LustreError: 101023:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2858.729299] LustreError: 101023:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2858.732017] LustreError: 101023:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2858.835667] Lustre: server umount lustre-OST0000 complete [ 2858.839478] LustreError: 101023:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2868.467535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2868.649586] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2868.657472] Lustre: Found index 0 for lustre-OST0000, updating log [ 2868.662187] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2868.754213] Lustre: lustre-OST0000: new disk, initializing [ 2868.757525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2868.769522] Lustre: 101558:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2869.141833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2875.608622] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2879.455816] Lustre: Failing over lustre-OST0000 [ 2879.609942] Lustre: server umount lustre-OST0000 complete [ 2884.582892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2884.584260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2884.613704] Lustre: Skipped 1 previous similar message [ 2890.100323] Lustre: server umount lustre-MDT0000 complete [ 2890.733620] LustreError: 98468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2890.755365] LustreError: 98468:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2894.297170] LustreError: 98454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790775783 with bad export cookie 7678838592011987065 [ 2894.311088] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2894.318701] LustreError: 98454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2894.612091] Lustre: server umount lustre-MDT0001 complete [ 2903.873171] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 09:43:10 (1790775790) [ 2920.668301] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 2932.947336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2933.604122] LustreError: 103903:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2933.869626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2940.404609] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2949.841859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2954.652841] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2958.113574] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2964.916584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2971.895562] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2981.316804] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2981.439131] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2981.533673] Lustre: lustre-OST0001: new disk, initializing [ 2981.538674] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2981.545524] Lustre: 106220:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2981.986709] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2981.989874] Lustre: Skipped 2 previous similar messages [ 2988.790150] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2991.744710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2991.752064] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2991.882135] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2997.330764] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3006.266928] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3012.066956] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3012.085365] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3012.109156] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3012.111957] Lustre: Skipped 3 previous similar messages [ 3017.717084] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3017.727097] Lustre: Skipped 5 previous similar messages [ 3025.376164] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3025.666055] Lustre: server umount lustre-MDT0001 complete [ 3027.944682] LustreError: 103903:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3027.962877] LustreError: 103903:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3036.712111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3036.897516] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3036.913989] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3037.003269] Lustre: 103892:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3038.177293] LustreError: 107748:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3038.195717] LustreError: 107748:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3038.211217] LustreError: 103895:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3038.353453] LustreError: 107764:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.104@tcp: cfg command failed: rc = -17 [ 3038.361410] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3038.361410] [ 3043.343643] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3043.465579] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3054.578899] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 09:45:41 (1790775941) [ 3073.379718] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 3092.901222] Lustre: 109837:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3119.864305] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3124.460368] Lustre: 110974:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3206.642247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3206.655446] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3206.676558] Lustre: Skipped 2 previous similar messages [ 3206.685104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3206.691961] Lustre: Skipped 5 previous similar messages [ 3210.175709] Lustre: server umount lustre-MDT0000 complete [ 3212.258434] LustreError: 103900:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3212.278567] LustreError: 103900:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 3213.495648] LustreError: 109839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790776102 with bad export cookie 7678838592011988059 [ 3213.498657] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3213.502221] LustreError: 109839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3213.713961] Lustre: server umount lustre-MDT0001 complete [ 3226.771168] Lustre: server umount lustre-OST0000 complete [ 3240.432014] Lustre: server umount lustre-OST0001 complete [ 3245.701051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3254.643715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3263.211753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3275.535650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3300.198094] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 3312.547543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3312.574239] Lustre: lustre-MDT0000: reset Object Index mappings [ 3313.192919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3313.195993] Lustre: Skipped 1 previous similar message [ 3318.546905] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3328.171557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3328.204901] Lustre: lustre-MDT0001: reset Object Index mappings [ 3333.893814] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3337.194691] Lustre: 116072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3344.172536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3352.326497] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3354.169670] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3358.285914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:350 to 0x280000401:386) [ 3361.638288] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3367.414712] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3367.421751] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3368.817341] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3377.851902] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3381.921289] Lustre: 117952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3432.358773] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 09:51:59 (1790776319) [ 3459.559636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3459.575903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3459.585817] Lustre: Skipped 5 previous similar messages [ 3459.604849] Lustre: Skipped 7 previous similar messages [ 3463.287587] Lustre: server umount lustre-MDT0000 complete [ 3466.996567] LustreError: 114905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790776355 with bad export cookie 7678838592012034910 [ 3467.003944] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3467.009346] LustreError: 114905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3469.795632] LustreError: 116126:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3469.809630] LustreError: 116126:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 3473.635314] Lustre: server umount lustre-MDT0001 complete [ 3492.320339] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3492.618949] Lustre: server umount lustre-OST0000 complete [ 3496.362343] LustreError: 93453:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff92280e9d3480 x1877764220585728/t0(0) o38->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3502.729856] Lustre: server umount lustre-OST0001 complete [ 3510.097767] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 3520.978411] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 3557.609915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3557.963178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3558.012296] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3558.221468] Lustre: lustre-MDT0000: new disk, initializing [ 3558.302689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3558.306773] Lustre: Skipped 3 previous similar messages [ 3558.321235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3563.460687] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3570.732935] Lustre: server umount lustre-MDT0000 complete [ 3579.943311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3580.424721] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3580.442430] Lustre: 122227:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3580.464954] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3586.420748] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3592.920580] Lustre: server umount lustre-MDT0000 complete [ 3609.030709] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 09:54:55 (1790776495) [ 3611.653427] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3614.053846] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 09:55:00 (1790776500) [ 3632.192343] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 3641.997354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3645.842934] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3654.993192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3655.170301] Lustre: 124277:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3655.251561] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3655.255592] Lustre: Skipped 1 previous similar message [ 3655.312360] Lustre: lustre-MDT0001: new disk, initializing [ 3655.487328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3655.503530] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3661.773405] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3666.681700] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3672.556709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3672.811663] Lustre: lustre-OST0000: new disk, initializing [ 3672.814991] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3672.822083] Lustre: 125853:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3673.956189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3673.970745] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3674.163459] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3678.826526] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3689.420717] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3689.573656] Lustre: lustre-OST0001: new disk, initializing [ 3689.579845] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3689.588524] Lustre: 126724:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3690.884858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3690.895273] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3690.978080] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3697.592261] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3708.318636] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3712.249172] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4087.891697] Lustre: 133146:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4087.898111] Lustre: 133146:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4087.903763] Lustre: 133146:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4087.910831] Lustre: 133146:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4087.921877] Lustre: 133146:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4087.926788] Lustre: 133146:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4316.774206] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:06:42 (1790777202) [ 4336.106181] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4336.129572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4336.158522] Lustre: Skipped 4 previous similar messages [ 4336.176888] Lustre: Skipped 7 previous similar messages [ 4339.378928] Lustre: server umount lustre-MDT0000 complete [ 4341.228887] LustreError: 124284:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4341.254097] LustreError: 124284:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 4343.331975] LustreError: 124270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790777232 with bad export cookie 7678838592012102243 [ 4343.341900] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4343.348445] LustreError: 124270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4346.338321] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4346.342319] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4346.358822] Lustre: Skipped 3 previous similar messages [ 4346.376322] Lustre: Skipped 1 previous similar message [ 4349.836885] Lustre: server umount lustre-MDT0001 complete [ 4359.537237] Lustre: server umount lustre-OST0000 complete [ 4368.715979] Lustre: server umount lustre-OST0001 complete [ 4377.309873] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 4389.814176] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 4428.379803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4428.691183] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4428.779863] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4428.882828] Lustre: lustre-MDT0000: new disk, initializing [ 4428.986527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4428.992746] Lustre: Skipped 5 previous similar messages [ 4429.010095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4433.678516] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4443.599025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4443.736759] Lustre: 136194:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4443.784275] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4443.787755] Lustre: Skipped 1 previous similar message [ 4443.851803] Lustre: lustre-MDT0001: new disk, initializing [ 4443.980547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4443.993285] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4448.451941] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4457.819939] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4463.969516] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4470.939944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4471.357919] Lustre: lustre-OST0000: new disk, initializing [ 4471.365231] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4471.382669] Lustre: 138162:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4471.506277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4471.514839] Lustre: Skipped 1 previous similar message [ 4472.439881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4472.453513] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4472.587562] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4477.898676] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4488.247770] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4499.179624] Lustre: lustre-MDD0000: changelog on [ 4502.564742] Lustre: lustre-MDD0001: changelog on [ 6290.620903] Lustre: lustre-MDD0001: changelog off [ 6292.736103] Lustre: 136201:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6293.595858] Lustre: lustre-MDD0000: changelog off [ 6300.641616] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6300.641852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6300.655425] LustreError: Skipped 1 previous similar message [ 6300.657597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6300.672972] Lustre: Skipped 1 previous similar message [ 6300.685918] Lustre: Skipped 3 previous similar messages [ 6303.721171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6303.726646] Lustre: Skipped 1 previous similar message [ 6306.765804] Lustre: server umount lustre-OST0000 complete [ 6312.930370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6312.936252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6312.945027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6313.957993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6316.583130] Lustre: server umount lustre-MDT0000 complete [ 6319.079218] LustreError: 136201:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6319.098709] LustreError: 136201:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6320.643156] LustreError: 136187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790779209 with bad export cookie 7678838592012877906 [ 6320.649749] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6320.659545] LustreError: 136187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6321.290131] Lustre: server umount lustre-MDT0001 complete [ 6332.239669] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 6335.165913] Key type lgssc unregistered [ 6335.527838] LNet: 147287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6335.534882] LNetError: 147287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6336.551075] LNet: Removed LNI 192.168.201.104@tcp [ 6337.579263] Key type .llcrypt unregistered [ 6337.581273] Key type ._llcrypt unregistered [ 6358.527857] Key type ._llcrypt registered [ 6358.530236] Key type .llcrypt registered [ 6358.634684] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 6374.259638] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 6375.157378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6375.219926] alg: No test for adler32 (adler32-zlib) [ 6376.474286] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 6377.163236] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 6379.096576] Key type lgssc registered [ 6381.099841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6432.165777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6432.198955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6433.624489] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6433.646140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6433.801086] Lustre: lustre-MDT0000: new disk, initializing [ 6433.921533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6433.940103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6440.188707] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6454.663707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6454.947153] Lustre: 150507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6455.117293] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6455.122485] Lustre: Skipped 1 previous similar message [ 6455.236542] Lustre: lustre-MDT0001: new disk, initializing [ 6455.338926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6455.428092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6455.436342] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6460.414188] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6471.698383] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6478.947976] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6487.829104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6488.088104] Lustre: lustre-OST0000: new disk, initializing [ 6488.091534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6488.104571] Lustre: 152479:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6488.256212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6495.499506] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6497.490250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6497.518519] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6497.687501] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6508.362206] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6516.045556] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_module obdecho/obdecho [ 6518.190973] LustreError: 153482:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6518.198879] LustreError: 153482:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6520.491200] LustreError: 153579:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6533.602185] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6533.604521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6533.616415] Lustre: Skipped 2 previous similar messages [ 6533.628265] Lustre: Skipped 2 previous similar messages [ 6538.730674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6538.742581] Lustre: Skipped 2 previous similar messages [ 6539.974634] Lustre: server umount lustre-MDT0000 complete [ 6544.358457] LustreError: 150500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790779433 with bad export cookie 6388417649960902580 [ 6544.365705] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6544.374160] LustreError: 150500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6544.694836] Lustre: server umount lustre-MDT0001 complete [ 6559.289587] Lustre: server umount lustre-OST0000 complete [ 6583.797299] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 6601.173798] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 6645.338160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6645.602026] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6645.629229] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6645.751745] Lustre: lustre-MDT0000: new disk, initializing [ 6645.883932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6645.897813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6649.746642] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6661.177157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6661.292601] Lustre: 157320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6661.378299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6661.384689] Lustre: Skipped 1 previous similar message [ 6661.549291] Lustre: lustre-MDT0001: new disk, initializing [ 6661.735375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6661.793066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6661.810419] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6667.146445] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6678.829346] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6687.189428] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6698.587855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6699.295664] Lustre: lustre-OST0000: new disk, initializing [ 6699.310944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6699.326482] Lustre: 159286:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6699.454992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6704.761384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6704.771891] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6704.957076] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6707.039423] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6718.550498] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6725.474377] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6725.761795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6731.057211] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6731.282359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6735.845880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6735.857860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6735.861734] LustreError: Skipped 1 previous similar message [ 6735.866546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6735.932831] Lustre: Skipped 1 previous similar message [ 6740.382996] Lustre: server umount lustre-OST0000 complete [ 6746.082419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6746.087757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6746.128056] Lustre: Skipped 1 previous similar message [ 6746.136589] Lustre: Skipped 1 previous similar message [ 6751.203887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6751.214904] Lustre: Skipped 2 previous similar messages [ 6759.393366] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6759.749416] Lustre: server umount lustre-MDT0000 complete [ 6760.423299] LustreError: 157331:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6760.450039] LustreError: 157331:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6763.580049] LustreError: 157310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790779652 with bad export cookie 6388417649960903539 [ 6763.593440] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6764.213329] Lustre: server umount lustre-MDT0001 complete [ 6768.106300] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:47:34 (1790779654) [ 6769.735673] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6771.587507] Lustre: DEBUG MARKER: == conf-sanity test 138: MDT mount failure must not LBUG destroying recovery exports ========================================================== 10:47:38 (1790779658) [ 6792.013591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6792.931629] LustreError: 162265:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6793.054651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6798.323198] LustreError: 162266:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6798.357280] LustreError: 162266:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6799.549156] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6802.411090] LustreError: 162265:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6807.522150] LustreError: 162266:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6812.446442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6819.693109] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6829.526099] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6836.469551] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6844.991625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6845.439926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6845.445098] Lustre: Skipped 1 previous similar message [ 6852.654712] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6861.157918] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6872.335387] Lustre: *** cfs_fail_loc=718, val=0*** [ 6873.062890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6873.064862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6873.075959] Lustre: Skipped 2 previous similar messages [ 6873.085142] Lustre: Skipped 4 previous similar messages [ 6874.929186] Lustre: server umount lustre-MDT0000 complete [ 6877.327203] LustreError: 162261:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6877.356371] LustreError: 162261:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6879.137910] LustreError: 162244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790779767 with bad export cookie 6388417649960904421 [ 6879.148519] Lustre: *** cfs_fail_loc=718, val=0*** [ 6879.155311] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6879.162537] LustreError: 162244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6879.201491] Lustre: Skipped 3 previous similar messages [ 6879.691929] Lustre: server umount lustre-MDT0001 complete [ 6890.348800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6899.680305] Lustre: 148719:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790779772/real 1790779772] req@00000000b385b7ce x1877768158698624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790779788 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6899.692247] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6900.704777] Lustre: 148719:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790779773/real 1790779773] req@00000000e9f8d56f x1877768158698752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790779789 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6904.094265] Lustre: *** cfs_fail_loc=723, val=0*** [ 6904.098939] Lustre: Skipped 1 previous similar message [ 6904.219058] LustreError: 165620:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-122) [ 6904.230325] LustreError: 165620:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.104@tcp: cfg command failed: rc = -122 [ 6904.243509] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6904.243509] [ 6904.259438] LustreError: MGC192.168.201.104@tcp: Confguration from log lustre-MDT0000 failed from MGS -122. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6904.280111] LustreError: 165590:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -122 [ 6904.289646] LustreError: 165590:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -122 [ 6904.308370] LustreError: 165590:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 6904.311455] LustreError: 165590:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 6904.438222] Lustre: server umount lustre-MDT0000 complete [ 6904.443866] LustreError: 165590:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -122 [ 6904.785104] Lustre: 148722:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790779777/real 1790779777] req@000000009b7c819a x1877768158698880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790779793 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6909.472452] Lustre: 148721:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790779782/real 1790779782] req@00000000d570e5d8 x1877768158699136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790779798 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6913.525907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6913.886945] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6913.911951] LustreError: 162256:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 6914.235337] LustreError: 164098:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6914.253167] LustreError: 164098:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6914.311321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6915.360753] Lustre: 148719:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790779787/real 1790779787] req@00000000e0f52d7c x1877768158699264/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790779803 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6919.892107] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6928.952950] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6929.895031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6930.559762] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6932.389901] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6935.545115] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6935.579776] Lustre: lustre-MDT0001: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6935.802678] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6944.950678] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6946.952378] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6953.672400] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6955.663412] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6970.849494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6970.863626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6970.893731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6983.140224] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6983.333355] Lustre: server umount lustre-OST0000 complete [ 6991.842285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6991.842738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6991.877667] Lustre: Skipped 1 previous similar message [ 6993.781063] Lustre: server umount lustre-MDT0000 complete [ 6996.975563] LustreError: 166028:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6997.010282] LustreError: 166028:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 6997.955344] LustreError: 162244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790779886 with bad export cookie 6388417649960905373 [ 6997.970501] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6997.976449] LustreError: 162244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6998.466300] Lustre: server umount lustre-MDT0001 complete [ 7012.833974] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 7016.449918] Key type lgssc unregistered [ 7016.753631] LNet: 168853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7016.762456] LNetError: 168853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7017.830376] LNet: Removed LNI 192.168.201.104@tcp [ 7019.114993] Key type .llcrypt unregistered [ 7019.119624] Key type ._llcrypt unregistered [ 7042.321589] Key type ._llcrypt registered [ 7042.335335] Key type .llcrypt registered [ 7042.481864] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 7043.587335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7043.631779] alg: No test for adler32 (adler32-zlib) [ 7045.050922] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 7045.531186] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 7047.464553] Key type lgssc registered [ 7049.971068] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7062.615354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7062.658901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7064.503233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7070.518712] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7074.796804] LustreError: 170184:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7074.809658] LustreError: 170184:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7078.893660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7079.181273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7082.817830] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7091.089741] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7098.467461] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7107.616796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7108.101781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7110.210367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7116.405579] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7125.459661] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7135.713350] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7135.726583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7135.751078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7138.788484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7138.796749] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7138.988208] LustreError: 170181:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7139.163983] Lustre: server umount lustre-MDT0000 complete [ 7143.198925] LustreError: 170165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790780032 with bad export cookie 15429294174741998390 [ 7143.205228] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7143.217334] LustreError: 170165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7143.906124] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7143.908666] LustreError: 172025:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7143.935652] Lustre: Skipped 1 previous similar message [ 7143.978023] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7143.990773] Lustre: Skipped 2 previous similar messages [ 7149.026169] LustreError: 170180:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7149.027122] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7149.073798] Lustre: Skipped 1 previous similar message [ 7149.074563] LustreError: 170180:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7149.813590] Lustre: server umount lustre-MDT0001 complete [ 7157.903586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7174.222682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7174.795345] LustreError: 174031:0:(llog.c:1655:llog_backup()) MGC192.168.201.104@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7174.806334] Lustre: 174031:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.104@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7178.727466] LustreError: 174036:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7178.762762] LustreError: 174036:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7188.961897] LustreError: 174038:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7188.972390] LustreError: 174038:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7205.516119] LustreError: 174037:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7205.562838] LustreError: 174037:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 7235.554257] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7235.564035] Lustre: 174031:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.104@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7239.146515] LustreError: 174037:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7239.160037] LustreError: 174037:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 22 previous similar messages [ 7276.558355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7281.305294] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7281.650235] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7291.019556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7304.184062] LustreError: 174038:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7304.187407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7304.215808] LustreError: 174038:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 47 previous similar messages [ 7313.849975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7313.935706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 7319.031079] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7320.905729] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7330.507926] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7332.302036] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7339.001183] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7340.839896] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7359.971984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7359.973194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7359.975488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7359.975495] Lustre: Skipped 1 previous similar message [ 7360.013860] Lustre: Skipped 2 previous similar messages [ 7363.826980] Lustre: server umount lustre-MDT0000 complete [ 7368.156468] LustreError: 172769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790780256 with bad export cookie 15429294174742005096 [ 7368.161089] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7368.169441] LustreError: 172769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7368.580935] Lustre: server umount lustre-MDT0001 complete [ 7382.638331] Lustre: server umount lustre-OST0000 complete [ 7405.984901] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 7417.419070] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 7457.214899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7457.563625] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7457.595385] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7457.721139] Lustre: lustre-MDT0000: new disk, initializing [ 7457.849841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7457.869962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7464.616816] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7475.615756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7475.753758] Lustre: 179141:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7475.806511] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7475.810384] Lustre: Skipped 1 previous similar message [ 7475.881244] Lustre: lustre-MDT0001: new disk, initializing [ 7475.983101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7476.042386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7476.057426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7480.668282] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7490.666708] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7496.709427] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7503.579150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7503.991898] Lustre: lustre-OST0000: new disk, initializing [ 7503.998281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7504.004422] Lustre: 181110:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7509.615933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7509.626918] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7509.763314] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7511.662382] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7522.047413] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7527.136726] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7527.282474] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7532.875567] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7533.158811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7535.588355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7535.599124] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7535.601739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7535.601748] Lustre: Skipped 2 previous similar messages [ 7535.623563] Lustre: Skipped 1 previous similar message [ 7535.656744] LustreError: Skipped 1 previous similar message [ 7541.265362] Lustre: server umount lustre-OST0000 complete [ 7545.835438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7551.381925] Lustre: server umount lustre-MDT0000 complete [ 7555.481609] LustreError: 179135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790780444 with bad export cookie 15429294174742022120 [ 7555.490214] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7555.495408] LustreError: 179135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7555.751924] Lustre: server umount lustre-MDT0001 complete [ 7564.138266] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:00:51 (1790780451) [ 7571.028370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7571.399636] LustreError: 183524:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7571.412232] LustreError: 183524:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 7571.488642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7571.498742] Lustre: Skipped 1 previous similar message [ 7576.621575] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7585.805616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7591.592374] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7599.640707] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7605.688665] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7612.340188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7619.966570] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7628.567805] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7642.926534] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:02:09 (1790780529) [ 7647.715894] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7647.724380] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7647.740576] Lustre: Skipped 1 previous similar message [ 7647.756095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7647.770521] Lustre: Skipped 7 previous similar messages [ 7653.463716] Lustre: server umount lustre-OST0000 complete [ 7663.002432] Lustre: server umount lustre-MDT0000 complete [ 7666.545581] LustreError: 185379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790780555 with bad export cookie 15429294174742023002 [ 7666.551066] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7666.562971] LustreError: 185379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7666.925389] Lustre: server umount lustre-MDT0001 complete [ 7675.951503] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 7678.437778] Key type lgssc unregistered [ 7678.789496] LNet: 187262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7678.794814] LNetError: 187262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7679.848441] LNet: Removed LNI 192.168.201.104@tcp [ 7681.117530] Key type .llcrypt unregistered [ 7681.120111] Key type ._llcrypt unregistered [ 7700.631709] Key type ._llcrypt registered [ 7700.634231] Key type .llcrypt registered [ 7700.725316] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 7701.906735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7701.931082] alg: No test for adler32 (adler32-zlib) [ 7703.009093] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 7703.151901] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 7704.818422] Key type lgssc registered [ 7706.059599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7712.856063] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7712.880049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7728.480704] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7733.600527] LustreError: 188520:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.104@tcp: failed processing log, type 4: rc = -110 [ 7738.721363] LustreError: 188520:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7738.736732] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7738.746912] LustreError: MGC192.168.201.104@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7738.752989] LustreError: 188520:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7738.759907] LustreError: 188520:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 7738.765786] LustreError: 188520:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 7738.829742] Lustre: server umount lustre-OST0000 complete [ 7738.833332] LustreError: 188520:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7749.609972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7751.002580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7755.400238] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7760.440370] LustreError: 189022:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7760.484358] LustreError: 189022:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7763.586096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7764.193212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7770.536504] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7778.041641] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7784.007125] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7791.510918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7791.757875] LustreError: 190837:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7791.786072] Lustre: 190837:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.104@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7791.989695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7798.656345] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7805.897488] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7808.810624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7808.823385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7808.829819] Lustre: Skipped 1 previous similar message [ 7808.857744] Lustre: Skipped 2 previous similar messages [ 7814.122038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7814.141756] Lustre: Skipped 2 previous similar messages [ 7819.234424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7819.243852] Lustre: Skipped 2 previous similar messages [ 7822.304510] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7822.668686] Lustre: server umount lustre-MDT0000 complete [ 7824.362040] LustreError: 191600:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7824.392774] LustreError: 191600:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7826.643324] LustreError: 190865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790780715 with bad export cookie 4607248743954129019 [ 7826.645236] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7826.650783] LustreError: 190865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7827.260961] LustreError: 189751:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008bcd5221 x1877769549019776/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7827.485844] Lustre: server umount lustre-MDT0001 complete [ 7841.876508] Lustre: server umount lustre-OST0000 complete [ 7853.776096] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 7865.628343] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 7903.658395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7903.911855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7903.930442] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7903.988978] Lustre: lustre-MDT0000: new disk, initializing [ 7904.062277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7904.080434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7909.111047] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7920.221560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7920.327838] Lustre: 194368:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7920.384493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7920.387304] Lustre: Skipped 1 previous similar message [ 7920.483775] Lustre: lustre-MDT0001: new disk, initializing [ 7920.599612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7920.635748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7920.652776] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7925.699913] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7935.277914] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7941.459835] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7947.939968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7948.181611] Lustre: lustre-OST0000: new disk, initializing [ 7948.185529] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7948.190968] Lustre: 196333:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7948.253229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7954.162402] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7956.071135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7956.082885] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7956.203711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7965.234730] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7970.435577] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7970.631542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7975.228236] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7975.583146] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7982.060901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7982.071359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7982.081292] Lustre: Skipped 2 previous similar messages [ 7982.083964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7982.094842] LustreError: Skipped 1 previous similar message [ 7987.176095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7987.184947] Lustre: Skipped 2 previous similar messages [ 7992.288173] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7992.517316] Lustre: server umount lustre-OST0000 complete [ 7997.410104] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7997.421324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7997.423238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7997.423246] Lustre: Skipped 2 previous similar messages [ 7997.442775] Lustre: Skipped 1 previous similar message [ 8002.442450] Lustre: server umount lustre-MDT0000 complete [ 8002.532488] LustreError: 194378:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8002.568727] LustreError: 194378:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8006.345161] LustreError: 195161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790780895 with bad export cookie 4607248743954129740 [ 8006.352513] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8006.357927] LustreError: 195161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8006.965031] Lustre: server umount lustre-MDT0001 complete [ 8015.488983] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:08:22 (1790780902) [ 8033.765541] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 8036.464700] Key type lgssc unregistered [ 8036.725640] LNet: 199280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8036.737650] LNetError: 199280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8036.758739] LNet: Removed LNI 192.168.201.104@tcp [ 8037.793167] Key type .llcrypt unregistered [ 8037.794538] Key type ._llcrypt unregistered [ 8055.055601] Key type ._llcrypt registered [ 8055.061230] Key type .llcrypt registered [ 8055.171689] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 8056.238762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8056.323484] alg: No test for adler32 (adler32-zlib) [ 8057.547842] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 8057.795120] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 8059.448311] Key type lgssc registered [ 8060.887885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8069.379976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8069.397039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8071.064754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8075.819134] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8080.356450] LustreError: 200514:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8080.385927] LustreError: 200514:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8085.136876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8085.484384] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8085.716632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8090.117805] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8098.542057] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8104.254713] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8112.545148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8112.848930] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8112.855346] LustreError: 202380:0:(llog.c:1696:llog_backup()) MGC192.168.201.104@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8112.862320] Lustre: 202380:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.104@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8113.042523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8120.322095] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8129.308683] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8132.066888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8132.075815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8132.091250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8135.650943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8135.657045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8135.684466] Lustre: Skipped 1 previous similar message [ 8138.319732] Lustre: server umount lustre-MDT0000 complete [ 8140.780763] LustreError: 200509:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8140.794895] LustreError: 200509:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8142.521057] LustreError: 200495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781031 with bad export cookie 13000301086978561575 [ 8142.528789] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8142.536641] LustreError: 200495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8142.922542] Lustre: server umount lustre-MDT0001 complete [ 8156.517319] Lustre: server umount lustre-OST0000 complete [ 8167.083229] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 8178.891634] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 8220.800709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8221.039757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8221.064180] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8221.144416] Lustre: lustre-MDT0000: new disk, initializing [ 8221.210380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8221.223237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8226.194152] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8237.531278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8237.703604] Lustre: 205908:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8237.760492] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8237.763414] Lustre: Skipped 1 previous similar message [ 8237.766320] LustreError: 206690:0:(llog.c:1696:llog_backup()) MGC192.168.201.104@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8237.770856] LustreError: 206690:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8237.777352] Lustre: 206690:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.104@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8237.809293] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8237.815126] Lustre: Skipped 1 previous similar message [ 8237.921587] Lustre: lustre-MDT0001: new disk, initializing [ 8238.090943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8238.158569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8238.177170] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8244.688959] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8257.705760] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8263.399351] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8269.285039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8269.610518] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8269.616264] Lustre: Skipped 1 previous similar message [ 8269.619590] LustreError: 207853:0:(llog.c:1696:llog_backup()) MGC192.168.201.104@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8269.627508] LustreError: 207853:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8269.638783] Lustre: 207853:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.104@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8269.647537] Lustre: 207853:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8269.675967] Lustre: lustre-OST0000: new disk, initializing [ 8269.680564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8269.686849] Lustre: 207878:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8269.819106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8276.827716] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8277.105206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8277.119210] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8277.281480] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8286.700486] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8291.897462] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8292.144229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8296.825075] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8297.115121] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8303.076916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8303.082659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8303.091485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8303.091492] Lustre: Skipped 2 previous similar messages [ 8303.132642] Lustre: Skipped 1 previous similar message [ 8308.204333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8308.215775] Lustre: Skipped 2 previous similar messages [ 8313.314367] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8313.484134] Lustre: server umount lustre-OST0000 complete [ 8318.433764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8318.440552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8318.449813] Lustre: Skipped 1 previous similar message [ 8318.455564] Lustre: Skipped 2 previous similar messages [ 8323.981622] Lustre: server umount lustre-MDT0000 complete [ 8328.181618] LustreError: 205901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781216 with bad export cookie 13000301086978562296 [ 8328.187423] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8328.191852] LustreError: 205901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8328.603800] Lustre: server umount lustre-MDT0001 complete [ 8337.472610] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:13:44 (1790781224) [ 8354.832409] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 8365.075665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8365.513549] LustreError: 211005:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8365.635399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8370.657587] LustreError: 211006:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8370.968128] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8375.780297] LustreError: 211005:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8380.583847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8385.620905] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8388.683497] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8396.016281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8402.770603] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8412.470047] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8412.679157] Lustre: lustre-OST0001: new disk, initializing [ 8412.681554] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8412.685879] Lustre: 213315:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8412.798804] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8412.805817] Lustre: Skipped 2 previous similar messages [ 8420.867685] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8422.482970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8422.502793] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8422.637608] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8433.334379] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8439.496441] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8441.743477] Lustre: DEBUG MARKER: ADD OST3 [ 8446.821691] loop: module loaded [ 8446.914374] loop0: detected capacity change from 0 to 204800000 [ 8447.200517] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8447.211301] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8447.226539] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8453.885685] loop0: detected capacity change from 0 to 204800000 [ 8454.294507] Lustre: DEBUG MARKER: STOP OST3 [ 8459.392555] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8459.715661] Lustre: lustre-OST0003: new disk, initializing [ 8459.721560] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8459.733538] Lustre: 215351:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8461.327493] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8461.461508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8461.473247] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8461.524654] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8462.942645] Lustre: Failing over lustre-OST0003 [ 8463.208882] Lustre: server umount lustre-OST0003 complete [ 8463.393287] LustreError: 215485:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8466.068823] LustreError: 212494:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8466.941146] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8466.964540] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8466.968348] Lustre: Skipped 1 previous similar message [ 8467.159217] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8477.050018] Lustre: DEBUG MARKER: START OST3 again [ 8482.292069] LustreError: 213222:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8482.301490] LustreError: 213222:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 8485.654019] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8486.155692] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8486.158859] Lustre: Skipped 1 previous similar message [ 8486.174919] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8486.525021] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8487.613567] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8487.613631] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8492.751787] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8515.154098] Lustre: Failing over lustre-OST0003 [ 8515.235852] Lustre: server umount lustre-OST0003 complete [ 8517.127904] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8517.149853] LustreError: 212492:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8517.168452] LustreError: 212492:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8534.498900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8534.519876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8534.534443] Lustre: Skipped 1 previous similar message [ 8534.541744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8534.554465] Lustre: Skipped 3 previous similar messages [ 8548.321972] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8548.738861] Lustre: server umount lustre-MDT0000 complete [ 8552.703483] LustreError: 210986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781441 with bad export cookie 13000301086978563178 [ 8552.708669] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8552.729257] LustreError: 210986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8567.264813] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8567.266839] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8567.289703] Lustre: Skipped 19 previous similar messages [ 8567.669843] Lustre: server umount lustre-MDT0001 complete [ 8571.451419] Lustre: server umount lustre-OST0000 complete [ 8575.110991] Lustre: server umount lustre-OST0001 complete [ 8582.443227] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 8590.839533] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 8627.420731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8627.716905] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8627.755762] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8627.844473] Lustre: lustre-MDT0000: new disk, initializing [ 8627.933457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8627.953979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8632.705481] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8643.131884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8643.260500] Lustre: 220769:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8643.478316] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8647.809140] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8657.306973] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8662.712479] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8669.000292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8669.234368] Lustre: lustre-OST0000: new disk, initializing [ 8669.237346] Lustre: Skipped 1 previous similar message [ 8669.240719] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8669.244379] Lustre: Skipped 2 previous similar messages [ 8669.246889] Lustre: 222738:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8671.175986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8671.194860] Lustre: Skipped 1 previous similar message [ 8671.199415] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8671.405941] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8675.033694] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8684.319847] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8689.932591] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8690.322447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8695.627321] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8695.933774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8701.931168] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8701.931561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8701.939923] LustreError: Skipped 1 previous similar message [ 8701.940670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8701.966281] Lustre: Skipped 6 previous similar messages [ 8704.402267] Lustre: server umount lustre-OST0000 complete [ 8710.116093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8714.756348] Lustre: server umount lustre-MDT0000 complete [ 8717.290727] LustreError: 220781:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8717.310756] LustreError: 220781:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 18 previous similar messages [ 8718.620368] LustreError: 220761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781607 with bad export cookie 13000301086978565467 [ 8718.624220] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8718.634741] LustreError: 220761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8718.912756] Lustre: server umount lustre-MDT0001 complete [ 8722.225623] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:20:09 (1790781609) [ 8740.012206] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 8752.064712] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 8784.060375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8784.291638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8784.318534] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8784.465287] Lustre: lustre-MDT0000: new disk, initializing [ 8784.573326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8789.290753] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8799.118958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8799.249334] Lustre: 227352:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8799.515335] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8803.439868] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8812.699093] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8818.891946] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8826.535754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8826.943424] Lustre: 229319:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8828.885503] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8833.152740] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8842.351671] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8847.105165] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8847.270441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8851.627186] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8851.807391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8854.501314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8854.514210] Lustre: Skipped 3 previous similar messages [ 8854.516734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8854.548199] Lustre: Skipped 5 previous similar messages [ 8859.869768] Lustre: server umount lustre-OST0000 complete [ 8870.209374] Lustre: server umount lustre-MDT0000 complete [ 8874.114791] LustreError: 228146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781762 with bad export cookie 13000301086978566349 [ 8874.118219] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8874.127295] LustreError: 228146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8874.382797] Lustre: server umount lustre-MDT0001 complete [ 8883.100663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8883.565663] LustreError: 231536:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8888.566672] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8897.017616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8897.389440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8897.392330] Lustre: Skipped 6 previous similar messages [ 8901.666399] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8910.246580] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8915.783373] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8923.562544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8930.863584] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8937.336693] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8947.690394] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8949.513130] Lustre: server umount lustre-OST0000 complete [ 8960.705364] Lustre: server umount lustre-MDT0000 complete [ 8964.568952] LustreError: 231517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781853 with bad export cookie 13000301086978567238 [ 8964.571246] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8964.589216] LustreError: 231517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8965.105363] Lustre: server umount lustre-MDT0001 complete [ 9078.324795] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 9081.109856] Key type lgssc unregistered [ 9081.341560] LNet: 235200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9081.346942] LNetError: 235200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9081.362968] LNet: Removed LNI 192.168.201.104@tcp [ 9082.113134] Key type .llcrypt unregistered [ 9082.115289] Key type ._llcrypt unregistered [ 9106.034817] Key type ._llcrypt registered [ 9106.037187] Key type .llcrypt registered [ 9106.125866] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 9124.178463] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 9125.730709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9125.775606] alg: No test for adler32 (adler32-zlib) [ 9126.898688] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 9127.139360] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 9128.809094] Key type lgssc registered [ 9130.339476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9170.019639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9170.091134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9171.388526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9171.443209] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9171.568541] Lustre: lustre-MDT0000: new disk, initializing [ 9171.711259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9171.734718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9176.251914] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9186.156360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9186.277616] Lustre: 238571:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9186.349668] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9186.355040] Lustre: Skipped 1 previous similar message [ 9186.470325] Lustre: lustre-MDT0001: new disk, initializing [ 9186.632662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9186.738109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9186.754991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9191.391602] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9201.182459] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9206.449813] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9212.938984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9213.192842] Lustre: lustre-OST0000: new disk, initializing [ 9213.195914] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9213.206754] Lustre: 240538:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9213.285248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9218.693338] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9222.745331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9222.752932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9222.886646] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9227.547092] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9232.398262] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9232.553954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9237.239605] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9237.453771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9243.618278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9243.630417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9243.652396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9243.665079] Lustre: Skipped 1 previous similar message [ 9245.651274] Lustre: server umount lustre-OST0000 complete [ 9253.346442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9253.360511] LustreError: Skipped 1 previous similar message [ 9253.368315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9253.392842] Lustre: Skipped 1 previous similar message [ 9253.402402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9255.644603] Lustre: server umount lustre-MDT0000 complete [ 9259.483927] LustreError: 240537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782148 with bad export cookie 3847420073572499679 [ 9259.489625] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9259.495243] LustreError: 240537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9259.999800] Lustre: server umount lustre-MDT0001 complete [ 9280.904213] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 9290.702875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9291.292512] LustreError: 243472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9291.321666] LustreError: 243472:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9291.401232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9296.870966] LustreError: 243473:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9297.030882] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9301.995814] LustreError: 243472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9305.751963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9306.233185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9311.958563] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9315.033068] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9322.498981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9322.803454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9330.316609] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9341.277750] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9341.593221] Lustre: lustre-OST0001: new disk, initializing [ 9341.598652] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9341.614821] Lustre: 245780:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9341.803971] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9347.355929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9347.374353] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9347.586138] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9351.813835] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9363.789807] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9372.219784] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9385.151879] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:31:11 (1790782271) [ 9388.003782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9388.017728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9388.037223] Lustre: Skipped 1 previous similar message [ 9388.052359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9388.062357] Lustre: Skipped 1 previous similar message [ 9393.611883] Lustre: server umount lustre-MDT0000 complete [ 9393.634504] LustreError: 243472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9393.646728] LustreError: 243472:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9397.951465] LustreError: 245779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782286 with bad export cookie 3847420073572500561 [ 9397.952619] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9397.959652] LustreError: 245779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9398.245635] Lustre: server umount lustre-MDT0001 complete [ 9412.810658] Lustre: server umount lustre-OST0000 complete [ 9414.625525] Lustre: 236783:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790782287/real 1790782287] req@ffff92290fd3ea00 x1877771042272256/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790782303 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9414.656664] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9414.668579] Lustre: Skipped 3 previous similar messages [ 9417.688414] Lustre: server umount lustre-OST0001 complete [ 9424.950381] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 9436.936139] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 9475.638095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9475.900591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9475.945889] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9476.042465] Lustre: lustre-MDT0000: new disk, initializing [ 9476.140908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9476.187463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9481.062672] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9492.130035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9492.242498] Lustre: 249727:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9492.295538] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9492.297942] Lustre: Skipped 1 previous similar message [ 9492.347123] Lustre: lustre-MDT0001: new disk, initializing [ 9492.465370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9492.475325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9496.788918] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9507.339787] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9513.016718] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9520.258656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9520.523793] Lustre: lustre-OST0000: new disk, initializing [ 9520.526600] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9520.531600] Lustre: 251696:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9521.943695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9521.956083] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9522.048970] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9527.773584] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9538.179372] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9542.911254] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9543.109678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9547.962383] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9548.214988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9552.867721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9552.867876] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9552.893766] LustreError: Skipped 1 previous similar message [ 9552.897708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9552.914502] Lustre: Skipped 1 previous similar message [ 9552.944957] Lustre: Skipped 4 previous similar messages [ 9556.651337] Lustre: server umount lustre-OST0000 complete [ 9563.107714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9563.135781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9563.145635] Lustre: Skipped 1 previous similar message [ 9566.574465] Lustre: server umount lustre-MDT0000 complete [ 9568.243386] LustreError: 249734:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9568.269844] LustreError: 249734:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9570.599266] LustreError: 249720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782459 with bad export cookie 3847420073572502696 [ 9570.600774] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9570.610387] LustreError: 249720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9571.118366] Lustre: server umount lustre-MDT0001 complete [ 9582.520488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9583.014963] LustreError: 253914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9583.092463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9583.100220] Lustre: Skipped 2 previous similar messages [ 9587.622624] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9597.847146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9604.183316] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9612.167655] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9619.181974] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9626.988234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9635.577926] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9644.115312] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9792.482161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9792.482798] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9792.491974] LustreError: Skipped 1 previous similar message [ 9792.496372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9792.506233] Lustre: Skipped 2 previous similar messages [ 9792.535856] Lustre: Skipped 1 previous similar message [ 9795.603432] Lustre: server umount lustre-OST0000 complete [ 9807.855306] LustreError: 253915:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9807.873500] LustreError: 253915:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9807.981219] Lustre: server umount lustre-MDT0000 complete [ 9813.238751] LustreError: 253896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782702 with bad export cookie 3847420073572503571 [ 9813.253604] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9813.266396] LustreError: 253896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9813.581964] Lustre: server umount lustre-MDT0001 complete [ 9914.851835] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 9917.701372] Key type lgssc unregistered [ 9918.036833] LNet: 257537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9918.045366] LNetError: 257537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9918.070469] LNet: Removed LNI 192.168.201.104@tcp [ 9918.904130] Key type .llcrypt unregistered [ 9918.905979] Key type ._llcrypt unregistered [ 9944.961569] Key type ._llcrypt registered [ 9944.962987] Key type .llcrypt registered [ 9945.118325] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 9961.661370] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 9963.134273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9963.323560] alg: No test for adler32 (adler32-zlib) [ 9964.431336] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 9964.731757] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 9966.432231] Key type lgssc registered [ 9967.747802] Lustre: Echo OBD driver; http://www.lustre.org/ [10022.742504] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [10034.678139] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10034.721120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10036.035825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10036.065569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10036.141926] Lustre: lustre-MDT0000: new disk, initializing [10036.231948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10036.249740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10041.636697] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10053.378070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10053.496887] Lustre: 261654:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10053.564714] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10053.569454] Lustre: Skipped 1 previous similar message [10053.640948] Lustre: lustre-MDT0001: new disk, initializing [10053.761049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10053.800189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10053.827798] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10058.220879] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10063.163919] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10070.775721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10071.209654] Lustre: lustre-OST0000: new disk, initializing [10071.220722] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10071.238113] Lustre: 263287:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10071.385965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10077.932495] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10078.271819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10078.294540] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10078.422896] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10089.196285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10089.430719] Lustre: lustre-OST0001: new disk, initializing [10089.435651] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10089.446655] Lustre: 264158:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10089.655763] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10096.651839] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10099.323042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10099.333763] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10099.449249] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10108.362647] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10115.864635] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10135.526189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10135.550293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10135.561337] Lustre: Skipped 1 previous similar message [10135.601842] Lustre: Skipped 3 previous similar messages [10137.533254] Lustre: server umount lustre-MDT0000 complete [10142.118146] LustreError: 262449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790783030 with bad export cookie 6216634047256413850 [10142.141824] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10142.738215] Lustre: server umount lustre-MDT0001 complete [10156.649096] Lustre: server umount lustre-OST0000 complete [10170.701510] Lustre: server umount lustre-OST0001 complete [10186.536653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10195.739471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10205.019487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10215.664887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10240.504550] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [10252.647063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10252.669041] Lustre: lustre-MDT0000: reset Object Index mappings [10253.280536] LustreError: 269168:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10253.301228] LustreError: 269168:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10253.390961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10257.371455] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10258.400884] LustreError: 269169:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10263.523354] LustreError: 269168:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10266.140807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10266.163756] Lustre: lustre-MDT0001: reset Object Index mappings [10266.659322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10271.360440] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10274.873042] Lustre: 270310:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10282.032153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10288.866681] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10291.580529] LustreError: 270664:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10296.816250] LustreError: 270665:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10297.611721] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10297.830822] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10297.835095] Lustre: Skipped 1 previous similar message [10298.879634] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10298.899028] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10305.069144] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10312.781557] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10316.602209] Lustre: 272184:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10322.916275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10322.926788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10322.940741] Lustre: Skipped 2 previous similar messages [10322.945092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10324.451109] LustreError: 269169:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10324.454364] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10324.486151] LustreError: 269169:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10324.512885] Lustre: Skipped 1 previous similar message [10325.216572] Lustre: server umount lustre-MDT0000 complete [10328.696718] LustreError: 269148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790783217 with bad export cookie 6216634047256420647 [10328.704411] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10328.713537] LustreError: 269148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10329.091667] Lustre: server umount lustre-MDT0001 complete [10342.631703] Lustre: server umount lustre-OST0000 complete [10345.444148] Lustre: 259120:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790783218/real 1790783218] req@ffff92290d7d1500 x1877771920995968/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790783234 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10345.482870] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10345.498983] Lustre: Skipped 1 previous similar message [10347.991277] Lustre: server umount lustre-OST0001 complete [10371.762537] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:47:38 (1790783258) [10390.806533] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [10407.577616] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [10452.484372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10452.859452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10452.891943] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10452.975973] Lustre: lustre-MDT0000: new disk, initializing [10453.033384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10453.057298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10459.189284] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10471.567187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10471.683703] Lustre: 276460:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10471.729298] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10471.731977] Lustre: Skipped 1 previous similar message [10471.781883] Lustre: lustre-MDT0001: new disk, initializing [10471.899671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10471.916394] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10476.434345] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10486.958711] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10495.632085] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10503.823487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10504.131382] Lustre: lustre-OST0000: new disk, initializing [10504.137726] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10504.149169] Lustre: 278429:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10505.858940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10505.865231] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10505.939059] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10511.089431] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10522.071664] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10526.743208] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10526.928465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10531.726587] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10532.064804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10536.937155] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10536.943301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10536.966731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10536.971290] Lustre: Skipped 1 previous similar message [10540.391935] Lustre: server umount lustre-OST0000 complete [10547.176764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10547.201339] Lustre: Skipped 1 previous similar message [10547.203667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10547.219606] Lustre: Skipped 1 previous similar message [10550.181392] Lustre: server umount lustre-MDT0000 complete [10552.290329] LustreError: 276467:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10552.309786] LustreError: 276467:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10554.053975] LustreError: 276454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790783442 with bad export cookie 6216634047256422740 [10554.060828] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10554.072244] LustreError: 276454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10554.598901] Lustre: server umount lustre-MDT0001 complete [10576.704384] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [10588.885709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10589.291756] LustreError: 281366:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10589.410753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10589.420220] Lustre: Skipped 2 previous similar messages [10594.582544] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10604.112986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10609.744205] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10612.793510] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10619.952380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10626.973834] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10635.402533] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10635.530743] Lustre: lustre-OST0001: new disk, initializing [10635.534060] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10635.539763] Lustre: 283675:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10636.806216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10636.831889] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10636.919592] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10641.781710] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10652.373665] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10656.879758] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10662.287422] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10662.392984] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10672.613570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10672.624043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10672.640559] Lustre: Skipped 3 previous similar messages [10672.648756] Lustre: Skipped 3 previous similar messages [10674.349875] Lustre: server umount lustre-MDT0000 complete [10677.740651] LustreError: 281362:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10677.780205] LustreError: 281362:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10678.066592] LustreError: 281347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790783566 with bad export cookie 6216634047256423622 [10678.077796] LustreError: 281347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10678.086396] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10678.608577] Lustre: server umount lustre-MDT0001 complete [10693.254607] Lustre: server umount lustre-OST0000 complete [10706.524521] Lustre: server umount lustre-OST0001 complete [10725.280350] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [10736.276437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10736.750766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10736.763490] Lustre: Skipped 3 previous similar messages [10741.146929] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10749.791950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10754.905562] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10757.709248] Lustre: 288159:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10764.568670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10771.740812] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10781.777307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10782.186475] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10787.870282] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10789.662177] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10798.222819] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10801.963688] Lustre: 290036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10806.932765] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10806.936180] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10819.719349] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 11:55:06 (1790783706) [10837.474597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10837.485064] LustreError: Skipped 1 previous similar message [10837.492635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10837.513223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10841.424540] Lustre: server umount lustre-MDT0000 complete [10844.131097] LustreError: 287019:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10844.159645] LustreError: 287019:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [10845.194714] LustreError: 286999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790783733 with bad export cookie 6216634047256430041 [10845.199860] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10845.206660] LustreError: 286999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10851.792437] Lustre: server umount lustre-MDT0001 complete [10861.411924] Lustre: server umount lustre-OST0000 complete [10865.646358] Lustre: server umount lustre-OST0001 complete [10873.183563] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [10884.215517] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [10929.970945] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [10941.169831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10941.453653] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10941.505833] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10941.651096] Lustre: lustre-MDT0000: new disk, initializing [10941.764915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10946.471801] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10956.906612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10957.107563] Lustre: 294567:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10957.468564] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10962.809829] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10973.945313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10974.246620] Lustre: 296199:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10975.884575] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10981.181016] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10991.311399] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10991.537416] Lustre: 297070:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10992.661487] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10992.664758] Lustre: Skipped 1 previous similar message [10992.741549] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10998.141455] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11008.584986] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11013.577706] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11013.596132] Lustre: Skipped 1 previous similar message [11032.977286] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 11:58:38 (1790783918) [11035.308753] Lustre: server umount lustre-MDT0000 complete [11038.661749] LustreError: 297835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790783927 with bad export cookie 6216634047256437958 [11038.674609] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11038.709485] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11038.730453] Lustre: Skipped 5 previous similar messages [11038.739472] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11038.742149] Lustre: Skipped 8 previous similar messages [11041.009347] Lustre: server umount lustre-MDT0001 complete [11045.008402] Lustre: server umount lustre-OST0000 complete [11048.663936] Lustre: server umount lustre-OST0001 complete [11057.277689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11057.924448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11057.937691] Lustre: Skipped 7 previous similar messages [11062.427704] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11071.297713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11076.598854] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11084.163901] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11090.274759] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11097.304598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11102.896519] LustreError: 301146:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11102.917267] LustreError: 301146:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [11104.727755] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11112.193114] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11127.661663] loop1: detected capacity change from 0 to 204800000 [11127.990109] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11128.017167] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11128.031353] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11133.682107] loop1: detected capacity change from 0 to 204800000 [11139.930552] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11140.023755] LustreError: 299306:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs25636: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11140.035327] LustreError: 299306:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs25636 [11140.040777] LustreError: 299306:0:(mgs_handler.c:517:mgs_target_reg()) fs25636-MDT0001: Error checking target index: rc = -13 [11140.049501] LustreError: fs25636-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11140.055366] LustreError: 303034:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [11140.061288] LustreError: 303034:0:(tgt_mount.c:2128:server_put_super()) no obd fs25636-MDT0001 [11140.065691] LustreError: 303034:0:(tgt_mount.c:116:server_deregister_mount()) fs25636-MDT0001 not registered [11140.094089] Lustre: server umount fs25636-MDT0001 complete [11140.096338] LustreError: 303034:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11150.672604] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11150.746614] Lustre: Setting parameter fs25636-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs25636-MDT0001 [11150.784523] Lustre: srv-fs25636-MDT0001: No data found on store. Initialize space. [11150.788635] Lustre: Skipped 4 previous similar messages [11150.849778] Lustre: fs25636-MDT0001: new disk, initializing [11150.852803] Lustre: Skipped 3 previous similar messages [11150.997150] Lustre: srv-fs25636-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11151.013987] Lustre: 303503:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs25636-MDT0001-mdtlov: can't create new llog: rc = -115 [11151.029145] LustreError: 303503:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs25636-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11155.729174] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11162.644840] Lustre: server umount fs25636-MDT0001 complete [11178.311807] loop1: detected capacity change from 0 to 204800000 [11185.135142] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11187.394854] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11188.308771] Lustre: 305082:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs25636-MDT0001-mdtlov: can't create new llog: rc = -115 [11188.322976] LustreError: 305082:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs25636-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11194.399221] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11199.828933] Lustre: server umount fs25636-MDT0001 complete [11209.870711] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.4@tcp (stopping) [11210.209707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11210.215222] LustreError: Skipped 1 previous similar message [11210.217969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11210.226037] Lustre: Skipped 3 previous similar messages [11216.036846] Lustre: server umount lustre-MDT0000 complete [11224.352413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11224.467647] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11230.249928] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11230.685621] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11239.474593] loop1: detected capacity change from 0 to 204800000 [11244.846881] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11245.260313] Lustre: 307529:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs25636-MDT0001-mdtlov: can't create new llog: rc = -115 [11245.270995] LustreError: 307529:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs25636-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11250.726044] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11255.950538] Lustre: server umount fs25636-MDT0001 complete [11286.503056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11286.504679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11286.537737] Lustre: Skipped 5 previous similar messages [11291.081096] Lustre: server umount lustre-OST0000 complete [11301.450765] Lustre: server umount lustre-MDT0000 complete [11305.437646] LustreError: 305573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790784194 with bad export cookie 6216634047256440646 [11305.449806] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11305.454045] LustreError: 305573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11305.764315] Lustre: server umount lustre-MDT0001 complete [11317.207592] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [11319.989989] Key type lgssc unregistered [11320.358170] LNet: 309872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11320.363890] LNetError: 309872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11321.387539] LNet: Removed LNI 192.168.201.104@tcp [11322.667282] Key type .llcrypt unregistered [11322.669156] Key type ._llcrypt unregistered [11350.239587] Key type ._llcrypt registered [11350.241553] Key type .llcrypt registered [11350.384224] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [11351.659962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11351.682338] alg: No test for adler32 (adler32-zlib) [11352.897806] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [11353.144303] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [11354.880539] Key type lgssc registered [11356.209517] Lustre: Echo OBD driver; http://www.lustre.org/ [11365.345230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11365.367280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11366.964433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11371.815501] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11377.139187] LustreError: 311159:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11377.158382] LustreError: 311159:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11380.622973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11381.497353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11387.601617] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11398.347584] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11405.853909] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11414.519858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11414.896767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11423.211776] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11427.197537] LustreError: 312991:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11432.432046] LustreError: 312993:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11435.088773] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11436.524657] LustreError: 312991:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11436.567837] LustreError: 312991:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11440.859068] LustreError: 312991:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11451.017954] LustreError: 313473:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11451.036501] LustreError: 313473:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11460.400572] loop1: detected capacity change from 0 to 204800000 [11460.547694] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11460.561220] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11460.571198] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11466.014320] loop1: detected capacity change from 0 to 204800000 [11467.239880] LustreError: 313473:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11467.275870] LustreError: 313473:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [11472.711776] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11472.947556] Lustre: fs5967-OST0000: new disk, initializing [11472.954146] Lustre: srv-fs5967-OST0000: No data found on store. Initialize space. [11472.970883] Lustre: 314899:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11473.108695] Lustre: fs5967-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11480.261662] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11489.362272] Lustre: server umount fs5967-OST0000 complete [11502.210430] LustreError: 312994:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11502.238845] LustreError: 312994:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 18 previous similar messages [11503.420499] loop1: detected capacity change from 0 to 204800000 [11508.198792] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11509.563180] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11509.836615] Lustre: fs5967-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11513.342391] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11518.052293] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11528.625813] loop2: detected capacity change from 0 to 204800000 [11528.994633] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11529.022585] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11529.046685] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11534.480386] loop2: detected capacity change from 0 to 204800000 [11540.150668] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11540.288749] LustreError: fs5967-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11540.295730] LustreError: fs5967-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11540.298416] LustreError: 317940:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [11540.309970] LustreError: 317940:0:(tgt_mount.c:2128:server_put_super()) no obd fs5967-OST0001 [11540.312948] LustreError: 317940:0:(tgt_mount.c:116:server_deregister_mount()) fs5967-OST0001 not registered [11540.405957] Lustre: server umount fs5967-OST0001 complete [11540.410541] LustreError: 317940:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11553.650440] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11553.743550] Lustre: fs5967-OST0001: new disk, initializing [11553.748481] Lustre: srv-fs5967-OST0001: No data found on store. Initialize space. [11553.760620] Lustre: 318491:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11553.853336] Lustre: fs5967-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11562.812367] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11568.792892] LustreError: 313472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11568.834449] LustreError: 313472:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [11572.964943] Lustre: server umount fs5967-OST0000 complete [11584.164978] Lustre: server umount fs5967-OST0001 complete [11597.576126] loop1: detected capacity change from 0 to 204800000 [11604.917590] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11605.362506] Lustre: fs5967-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11612.702250] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11620.247324] loop2: detected capacity change from 0 to 204800000 [11626.805578] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11634.788413] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11645.347903] Lustre: server umount fs5967-OST0000 complete [11654.782142] Lustre: server umount fs5967-OST0001 complete [11661.806677] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11666.944173] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11672.049671] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11672.063795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11672.081477] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11672.086894] Lustre: Skipped 1 previous similar message [11672.120490] Lustre: Skipped 1 previous similar message [11677.153988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11677.170849] Lustre: Skipped 1 previous similar message [11682.274655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11682.292933] Lustre: Skipped 1 previous similar message [11686.368386] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11686.682170] Lustre: server umount lustre-OST0000 complete [11692.517752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11692.525310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11692.529553] Lustre: Skipped 1 previous similar message [11697.643499] LustreError: 316781:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11697.667952] LustreError: 316781:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 65 previous similar messages [11697.859386] Lustre: server umount lustre-MDT0000 complete [11702.253526] LustreError: 311140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790784591 with bad export cookie 17990094962016570871 [11702.270783] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11702.672860] Lustre: server umount lustre-MDT0001 complete [11722.183103] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [11725.867351] Key type lgssc unregistered [11726.342579] LNet: 324437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11726.356196] LNetError: 324437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11726.401939] LNet: Removed LNI 192.168.201.104@tcp [11727.658394] Key type .llcrypt unregistered [11727.661199] Key type ._llcrypt unregistered [11765.975735] Key type ._llcrypt registered [11765.977834] Key type .llcrypt registered [11766.098407] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [11781.957773] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [11783.109614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11783.359385] alg: No test for adler32 (adler32-zlib) [11784.554803] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [11784.818437] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [11786.561302] Key type lgssc registered [11788.415769] Lustre: Echo OBD driver; http://www.lustre.org/ [11832.387188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11832.431612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11833.808731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11833.845151] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11833.937417] Lustre: lustre-MDT0000: new disk, initializing [11834.008261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11834.056882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11840.283528] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11851.393285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11851.557827] Lustre: 328318:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11851.670235] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11851.680103] Lustre: Skipped 1 previous similar message [11851.802377] Lustre: lustre-MDT0001: new disk, initializing [11851.924265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11852.013519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11852.038942] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11856.924395] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11867.757513] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11874.899686] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11883.763528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11884.088868] Lustre: lustre-OST0000: new disk, initializing [11884.101447] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11884.120708] Lustre: 330288:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11884.305963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11892.113779] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11893.383309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11893.402435] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11893.594071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11902.758328] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11905.426651] Lustre: Mounted lustre-client - version 2.17.59_4_ge4c1f18 [11906.560587] Lustre: Unmounted lustre-client [11919.330330] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11919.336284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11919.337954] LustreError: Skipped 1 previous similar message [11919.346757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11920.871551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11920.880152] Lustre: Skipped 2 previous similar messages [11923.289254] Lustre: server umount lustre-OST0000 complete [11929.061798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11929.066501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11929.096601] Lustre: Skipped 1 previous similar message [11929.104573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11929.117632] Lustre: Skipped 1 previous similar message [11933.634119] Lustre: server umount lustre-MDT0000 complete [11937.952262] LustreError: 328310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790784826 with bad export cookie 6298455472436003571 [11937.971263] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11937.979059] LustreError: 328310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11938.385301] Lustre: server umount lustre-MDT0001 complete [11954.043611] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [11957.487294] Key type lgssc unregistered [11966.112439] LNet: 1 local NIs in recovery (showing 1): 192.168.201.104@tcp [11967.137056] LNet: 332449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11967.153683] LNetError: 332449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12035.820079] LNet: Removed LNI 192.168.201.104@tcp [12036.956676] Key type .llcrypt unregistered [12036.959148] Key type ._llcrypt unregistered [12058.641403] Key type ._llcrypt registered [12058.644798] Key type .llcrypt registered [12058.812690] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [12079.094894] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [12080.849602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12080.922061] alg: No test for adler32 (adler32-zlib) [12082.106991] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [12082.411708] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [12084.191102] Key type lgssc registered [12085.973916] Lustre: Echo OBD driver; http://www.lustre.org/ [12119.505127] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 12:16:46 (1790785006) [12128.697430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12128.728286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12130.266687] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12130.379510] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12130.534841] Lustre: lustre-MDT0000: new disk, initializing [12130.643942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12130.672329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12137.103573] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12147.544474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12147.662894] Lustre: 335691:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12147.746413] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12147.753239] Lustre: Skipped 1 previous similar message [12147.868189] Lustre: lustre-MDT0001: new disk, initializing [12148.004844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12148.084098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12148.093152] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12152.440527] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12160.910266] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12166.317316] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12173.741796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12174.107380] Lustre: lustre-OST0000: new disk, initializing [12174.110442] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12174.116685] Lustre: 337659:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12174.201161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12179.594088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12179.608428] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12179.696755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12181.110446] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12191.209286] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12210.657741] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12210.659476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12210.667978] LustreError: Skipped 1 previous similar message [12210.673625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12210.703590] Lustre: Skipped 1 previous similar message [12213.129207] Lustre: server umount lustre-OST0000 complete [12219.873950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12219.892631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12219.919843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12219.938140] Lustre: Skipped 1 previous similar message [12220.927585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12220.950752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12220.958583] Lustre: Skipped 1 previous similar message [12226.020234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12231.137802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12231.151605] Lustre: Skipped 2 previous similar messages [12232.160555] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12232.504772] Lustre: server umount lustre-MDT0000 complete [12237.318351] LustreError: 336485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790785126 with bad export cookie 832360517423818972 [12237.327774] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12237.340910] LustreError: 336485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12237.711397] Lustre: server umount lustre-MDT0001 complete [12251.196878] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [12254.669819] Key type lgssc unregistered [12255.036765] LNet: 339567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12255.057161] LNetError: 339567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12255.098302] LNet: Removed LNI 192.168.201.104@tcp [12256.527974] Key type .llcrypt unregistered [12256.529982] Key type ._llcrypt unregistered [12282.020935] Key type ._llcrypt registered [12282.024452] Key type .llcrypt registered [12282.134488] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [12283.601066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12283.617750] alg: No test for adler32 (adler32-zlib) [12284.843217] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [12285.126685] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [12286.849750] Key type lgssc registered [12288.171636] Lustre: Echo OBD driver; http://www.lustre.org/ [12296.780509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12296.807707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12298.515598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12304.296449] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12308.965577] LustreError: 340855:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12308.981862] LustreError: 340855:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12313.026755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12313.448083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12318.889755] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12326.743958] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12332.480471] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12338.871049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12339.220749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12345.049507] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12355.642037] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12367.277767] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12369.528738] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12375.009816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12375.024433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12375.066831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12378.272707] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.4@tcp (stopping) [12379.109629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12379.140164] Lustre: Skipped 1 previous similar message [12380.450761] Lustre: server umount lustre-MDT0000 complete [12383.355062] LustreError: 340852:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12385.196159] LustreError: 340835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790785273 with bad export cookie 17115008015056097144 [12385.201270] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12385.213118] LustreError: 340835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12388.480512] LustreError: 340850:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12388.502500] LustreError: 340850:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12388.510861] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.4@tcp (stopping) [12388.531561] Lustre: Skipped 3 previous similar messages [12389.352247] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12391.813123] Lustre: server umount lustre-MDT0001 complete [12396.087471] Lustre: server umount lustre-OST0000 complete [12405.337090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12411.002615] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12413.971989] Lustre: 345039:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.104@tcp [12416.394645] Lustre: server umount MGS complete [12425.987115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12426.392799] LustreError: 345611:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12426.422106] LustreError: 345611:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12426.494277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12431.688190] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12431.843829] LustreError: 345612:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12431.866964] LustreError: 345612:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12442.066825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12442.084542] LustreError: 345611:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12442.111046] LustreError: 345611:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12442.638235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12447.642737] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12456.733602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12457.066606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12463.676262] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12471.904600] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12473.579436] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12479.383852] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12481.211525] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12487.143333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12487.152755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12487.165750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12487.171729] Lustre: Skipped 2 previous similar messages [12488.190414] Lustre: server umount lustre-OST0000 complete [12492.261781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12492.272600] Lustre: Skipped 1 previous similar message [12492.276879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12492.289136] Lustre: Skipped 2 previous similar messages [12494.704201] Lustre: server umount lustre-MDT0000 complete [12497.377920] LustreError: 346600:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12497.405766] LustreError: 346600:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12498.884983] LustreError: 345592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790785387 with bad export cookie 17115008015056097984 [12498.896937] LustreError: 345592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12498.903257] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12499.385352] Lustre: server umount lustre-MDT0001 complete [12512.042811] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [12515.247749] Key type lgssc unregistered [12515.637996] LNet: 349048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12515.646791] LNetError: 349048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12515.675698] LNet: Removed LNI 192.168.201.104@tcp [12516.737895] Key type .llcrypt unregistered [12516.754508] Key type ._llcrypt unregistered [12537.165992] Key type ._llcrypt registered [12537.173232] Key type .llcrypt registered [12537.348869] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [12554.507450] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [12555.496092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12555.675352] alg: No test for adler32 (adler32-zlib) [12557.138051] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [12557.458785] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [12559.176374] Key type lgssc registered [12560.692865] Lustre: Echo OBD driver; http://www.lustre.org/ [12603.779426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12603.811263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12605.211800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12605.255488] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12605.355370] Lustre: lustre-MDT0000: new disk, initializing [12605.438714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12605.470926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12611.484471] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12624.162413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12624.349666] Lustre: 352225:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12624.409670] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12624.413758] Lustre: Skipped 1 previous similar message [12624.501865] Lustre: lustre-MDT0001: new disk, initializing [12624.607957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12624.647771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12624.667991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12630.358219] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12640.457908] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12647.720375] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12656.534558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12656.871535] Lustre: lustre-OST0000: new disk, initializing [12656.880485] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12656.887231] Lustre: 354195:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12656.975333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12664.339757] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12664.861469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12664.874963] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12664.967510] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12675.031656] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12680.857426] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12681.199381] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12687.458182] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12687.751415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12690.404895] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12690.412960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12690.430636] Lustre: Skipped 1 previous similar message [12695.527382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12695.541442] Lustre: Skipped 2 previous similar messages [12696.373688] Lustre: server umount lustre-OST0000 complete [12701.669136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12701.680576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12701.727757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12705.764840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12705.765500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12705.804480] Lustre: Skipped 1 previous similar message [12707.230892] Lustre: server umount lustre-MDT0000 complete [12711.898653] LustreError: 352217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790785600 with bad export cookie 1889517712310322578 [12711.908893] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12711.921076] LustreError: 352217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12712.715893] Lustre: server umount lustre-MDT0001 complete [12724.105398] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 12:26:50 (1790785610) [12745.095720] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [12758.961476] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [12808.509594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12808.801730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12808.854967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12808.970128] Lustre: lustre-MDT0000: new disk, initializing [12809.149332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12809.175450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12815.123369] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12826.173594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12826.446185] Lustre: 358963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12826.661885] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12826.671334] Lustre: Skipped 1 previous similar message [12826.735700] Lustre: lustre-MDT0001: new disk, initializing [12826.851979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12826.896329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12826.908690] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12833.044773] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12845.116745] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12854.075592] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12863.491757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12863.902789] Lustre: lustre-OST0000: new disk, initializing [12863.907165] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12863.914400] Lustre: 360933:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12864.035496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12870.245492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12870.267260] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12870.391556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12871.258238] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12883.108429] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12926.945608] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12926.961205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12926.970801] LustreError: Skipped 1 previous similar message [12926.973102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12927.004535] Lustre: Skipped 1 previous similar message [12932.510781] Lustre: server umount lustre-OST0000 complete [12939.745614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12939.755460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12939.791367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12939.812917] Lustre: Skipped 3 previous similar messages [12944.699341] Lustre: server umount lustre-MDT0000 complete [12947.425198] LustreError: 361657:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12947.453319] LustreError: 361657:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12950.151084] LustreError: 358954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790785838 with bad export cookie 1889517712310323460 [12950.151659] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12950.160803] LustreError: 358954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12951.393906] Lustre: server umount lustre-MDT0001 complete [12967.623541] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [12971.902772] Key type lgssc unregistered [12972.255885] LNet: 363109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12972.260415] LNetError: 363109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12972.270064] LNet: Removed LNI 192.168.201.104@tcp [12972.999413] Key type .llcrypt unregistered [12973.001691] Key type ._llcrypt unregistered [12998.263491] Key type ._llcrypt registered [12998.266984] Key type .llcrypt registered [12998.404379] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [12999.873611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13000.093953] alg: No test for adler32 (adler32-zlib) [13001.060861] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [13001.265607] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [13002.952215] Key type lgssc registered [13004.702335] Lustre: Echo OBD driver; http://www.lustre.org/ [13014.537790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13014.555483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13016.120946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13021.822464] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [13025.315796] LustreError: 364402:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13025.338139] LustreError: 364402:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13030.381402] LustreError: 364403:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13031.993594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13032.514564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13039.418766] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [13047.599887] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13054.489932] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13063.067960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13063.737277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13070.771964] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [13079.957214] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13149.665590] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13149.672701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13149.707079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13151.714521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13151.718791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13151.732399] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13154.786845] Lustre: server umount lustre-OST0000 complete [13160.420494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13160.427711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13160.443885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13160.446520] Lustre: Skipped 1 previous similar message [13164.984653] Lustre: server umount lustre-MDT0000 complete [13167.079015] LustreError: 364403:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13167.090407] LustreError: 364403:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13169.694638] LustreError: 364384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790786058 with bad export cookie 8839195260025011819 [13169.715356] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13169.724388] LustreError: 364384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13170.109679] Lustre: server umount lustre-MDT0001 complete [13183.546647] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [13187.450375] Key type lgssc unregistered [13187.835605] LNet: 370013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13187.843041] LNetError: 370013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13188.906762] LNet: Removed LNI 192.168.201.104@tcp [13190.117900] Key type .llcrypt unregistered [13190.120302] Key type ._llcrypt unregistered [13305.864115] Key type ._llcrypt registered [13305.866842] Key type .llcrypt registered [13306.048310] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [13306.294743] Key type .llcrypt unregistered [13306.296873] Key type ._llcrypt unregistered [13396.113401] Key type ._llcrypt registered [13396.117021] Key type .llcrypt registered [13396.287782] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [13396.625357] Key type .llcrypt unregistered [13396.627653] Key type ._llcrypt unregistered [13434.850686] Key type ._llcrypt registered [13434.852882] Key type .llcrypt registered [13434.983363] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [13435.391419] Key type .llcrypt unregistered [13435.393167] Key type ._llcrypt unregistered [13471.648691] Key type ._llcrypt registered [13471.651910] Key type .llcrypt registered [13471.749468] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [13472.224355] Key type .llcrypt unregistered [13472.228082] Key type ._llcrypt unregistered [13557.134644] Key type ._llcrypt registered [13557.135968] Key type .llcrypt registered [13557.189737] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [13557.440329] Key type .llcrypt unregistered [13557.441882] Key type ._llcrypt unregistered [13594.694530] Key type ._llcrypt registered [13594.701162] Key type .llcrypt registered [13594.834241] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [13595.141558] Key type .llcrypt unregistered [13595.143306] Key type ._llcrypt unregistered [13630.236952] Key type ._llcrypt registered [13630.241572] Key type .llcrypt registered [13630.369711] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [13630.694576] Key type .llcrypt unregistered [13630.698132] Key type ._llcrypt unregistered [13664.050314] Key type ._llcrypt registered [13664.064183] Key type .llcrypt registered [13664.274110] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [13664.705597] Key type .llcrypt unregistered [13664.709493] Key type ._llcrypt unregistered [13700.475761] Key type ._llcrypt registered [13700.478734] Key type .llcrypt registered [13700.629681] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [13701.017679] Key type .llcrypt unregistered [13701.025274] Key type ._llcrypt unregistered [13738.901375] Key type ._llcrypt registered [13738.905981] Key type .llcrypt registered [13739.083516] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [13739.523391] Key type .llcrypt unregistered [13739.525529] Key type ._llcrypt unregistered [13809.712314] Key type ._llcrypt registered [13809.713824] Key type .llcrypt registered [13809.842483] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [13810.209314] Key type .llcrypt unregistered [13810.211310] Key type ._llcrypt unregistered [13842.713345] Key type ._llcrypt registered [13842.715652] Key type .llcrypt registered [13842.840147] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [13843.241409] Key type .llcrypt unregistered [13843.243235] Key type ._llcrypt unregistered [13878.286506] Key type ._llcrypt registered [13878.289500] Key type .llcrypt registered [13878.457200] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [13878.802339] Key type .llcrypt unregistered [13878.807843] Key type ._llcrypt unregistered [13936.326612] Key type ._llcrypt registered [13936.330254] Key type .llcrypt registered [13936.494338] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [13954.676419] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [13956.064509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13956.175075] alg: No test for adler32 (adler32-zlib) [13957.333438] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [13957.541996] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [13959.232531] Key type lgssc registered [13960.676763] Lustre: Echo OBD driver; http://www.lustre.org/ [13999.300057] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13999.377124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14000.707337] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14000.796262] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14000.973277] Lustre: tst250fs-MDT0000: new disk, initializing [14001.078828] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14001.102681] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14006.598990] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [14017.107039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14017.265510] Lustre: 390344:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14017.361409] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14017.368078] Lustre: Skipped 1 previous similar message [14017.418754] Lustre: tst250fs-MDT0001: Not available for connect from 0@lo (not set up) [14017.557438] Lustre: tst250fs-MDT0001: new disk, initializing [14017.717811] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14017.782340] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14017.803306] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14022.967370] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [14027.825049] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14033.565542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14033.847914] Lustre: tst250fs-OST0000: new disk, initializing [14033.850259] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14033.854198] Lustre: 391978:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14033.956827] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14034.610101] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14034.631533] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14034.769280] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14040.723426] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [14051.891969] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14052.108297] Lustre: tst250fs-OST0001: new disk, initializing [14052.112079] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14052.117364] Lustre: 392846:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14052.248442] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14059.450684] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [14060.707284] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14060.733051] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14060.829684] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14065.606804] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14067.930884] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14067.938195] Lustre: Skipped 1 previous similar message [14086.630366] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14086.638906] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14086.648802] Lustre: Skipped 3 previous similar messages [14086.658809] Lustre: Skipped 3 previous similar messages [14091.751028] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14091.760506] Lustre: Skipped 3 previous similar messages [14092.901733] Lustre: server umount tst250fs-MDT0000 complete [14096.529568] LustreError: 390336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790786985 with bad export cookie 13859924841998683020 [14096.541171] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14096.546901] LustreError: 390336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14096.870660] Lustre: tst250fs-MDT0001-lwp-OST0000: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14096.880826] Lustre: Skipped 1 previous similar message [14096.980139] Lustre: server umount tst250fs-MDT0001 complete [14103.177064] Lustre: server umount tst250fs-OST0000 complete [14107.582526] Lustre: server umount tst250fs-OST0001 complete [14126.067068] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [14135.329916] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [14166.407445] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:50:53 (1790787053) [14168.091451] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14183.953140] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [14194.552547] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [14228.521286] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13934 sec ========= 12:51:55 (1790787115) [14230.512328] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:51:57 (1790787117) === [14233.943535] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:52:00 (1790787120) === [14272.952294] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [14275.906192] Key type lgssc unregistered [14276.262748] LNet: 401990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14276.274678] LNetError: 401990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14276.291893] LNet: Removed LNI 192.168.201.104@tcp [14277.163778] Key type .llcrypt unregistered [14277.165414] Key type ._llcrypt unregistered