[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 474785175 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003149] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008841] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010017] pid_max: default: 32768 minimum: 301 [ 0.011151] LSM: Security Framework initializing [ 0.012066] Yama: becoming mindful. [ 0.013042] SELinux: Initializing. [ 0.014071] *** VALIDATE selinux *** [ 0.022123] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026607] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027158] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029114] *** VALIDATE tmpfs *** [ 0.030449] *** VALIDATE proc *** [ 0.032222] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.035275] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036174] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.042210] debug: unmapping init [mem 0xffffffffa6659000-0xffffffffa6660fff] [ 0.044180] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045714] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046026] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049016] ... value mask: 0000ffffffffffff [ 0.050019] ... max period: 00007fffffffffff [ 0.051019] ... fixed-purpose events: 3 [ 0.052011] ... event mask: 000000070000000f [ 0.053304] rcu: Hierarchical SRCU implementation. [ 0.055509] smp: Bringing up secondary CPUs ... [ 0.056578] x86: Booting SMP configuration: [ 0.057025] .... node #0, CPUs: #1 #2 #3 [ 0.060476] smp: Brought up 1 node, 4 CPUs [ 0.062015] smpboot: Max logical packages: 1 [ 0.063020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145618] node 0 deferred pages initialised in 80ms [ 0.149006] devtmpfs: initialized [ 0.150243] x86/mm: Memory block size: 128MB [ 0.152711] gcov: version magic: 0x41383552 [ 0.154672] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157124] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158343] pinctrl core: initialized pinctrl subsystem [ 0.160156] [ 0.160604] ************************************************************* [ 0.161010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163011] ** ** [ 0.165012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.167011] ** ** [ 0.168009] ** This means that this kernel is built to expose internal ** [ 0.170009] ** IOMMU data structures, which may compromise security on ** [ 0.172010] ** your system. ** [ 0.173010] ** ** [ 0.175010] ** If you see this message and you are not debugging the ** [ 0.177011] ** kernel, report this immediately to your vendor! ** [ 0.178010] ** ** [ 0.180014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182010] ************************************************************* [ 0.183583] NET: Registered protocol family 16 [ 0.185399] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.187052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.190062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.193171] cpuidle: using governor menu [ 0.194850] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.195713] PCI: Using configuration type 1 for base access [ 0.196142] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.204144] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.207064] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.211052] cryptd: max_cpu_qlen set to 1000 [ 0.214307] ACPI: Added _OSI(Module Device) [ 0.216021] ACPI: Added _OSI(Processor Device) [ 0.217013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.218026] ACPI: Added _OSI(Processor Aggregator Device) [ 0.223104] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.231305] ACPI: Interpreter enabled [ 0.232047] ACPI: PM: (supports S0 S3 S4 S5) [ 0.233015] ACPI: Using IOAPIC for interrupt routing [ 0.235107] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.239492] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.249804] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.252049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.255067] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.260112] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.265972] acpiphp: Slot [2] registered [ 0.267131] acpiphp: Slot [5] registered [ 0.268185] acpiphp: Slot [6] registered [ 0.269136] acpiphp: Slot [7] registered [ 0.271126] acpiphp: Slot [8] registered [ 0.272109] acpiphp: Slot [9] registered [ 0.273115] acpiphp: Slot [10] registered [ 0.274141] acpiphp: Slot [3] registered [ 0.275077] acpiphp: Slot [4] registered [ 0.277210] acpiphp: Slot [11] registered [ 0.278135] acpiphp: Slot [12] registered [ 0.280091] acpiphp: Slot [13] registered [ 0.280908] acpiphp: Slot [14] registered [ 0.282104] acpiphp: Slot [15] registered [ 0.283102] acpiphp: Slot [16] registered [ 0.285176] acpiphp: Slot [17] registered [ 0.286113] acpiphp: Slot [18] registered [ 0.288089] acpiphp: Slot [19] registered [ 0.289106] acpiphp: Slot [20] registered [ 0.291114] acpiphp: Slot [21] registered [ 0.293200] acpiphp: Slot [22] registered [ 0.295131] acpiphp: Slot [23] registered [ 0.296175] acpiphp: Slot [24] registered [ 0.298136] acpiphp: Slot [25] registered [ 0.299146] acpiphp: Slot [26] registered [ 0.301411] acpiphp: Slot [27] registered [ 0.303106] acpiphp: Slot [28] registered [ 0.304128] acpiphp: Slot [29] registered [ 0.306135] acpiphp: Slot [30] registered [ 0.307108] acpiphp: Slot [31] registered [ 0.309190] PCI host bridge to bus 0000:00 [ 0.311022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.313027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.315028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.318029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.321025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.325029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.327203] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.331512] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.334474] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.348020] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.354062] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.356020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.359021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.363019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.365539] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.368832] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.373051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.375906] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.382018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.396014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.402017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.409799] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.421015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.427014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.446017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.464000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.473014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.479019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.503022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.516420] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.529023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.537026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.554030] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.568114] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.580019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.592018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.608017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.620199] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.635026] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.648175] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.677029] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.692327] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.708020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.720021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.742016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.758726] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.761463] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.765080] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.767437] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.770289] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.777053] iommu: Default domain type: Passthrough [ 0.779579] SCSI subsystem initialized [ 0.781222] ACPI: bus type USB registered [ 0.783132] usbcore: registered new interface driver usbfs [ 0.785113] usbcore: registered new interface driver hub [ 0.788165] usbcore: registered new device driver usb [ 0.791277] pps_core: LinuxPPS API ver. 1 registered [ 0.793014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.796084] PTP clock support registered [ 0.798111] EDAC MC: Ver: 3.0.0 [ 0.801149] PCI: Using ACPI for IRQ routing [ 0.802631] NetLabel: Initializing [ 0.803000] NetLabel: domain hash size = 128 [ 0.806017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.809086] NetLabel: unlabeled traffic allowed by default [ 0.812128] vgaarb: loaded [ 0.814274] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.816016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.824741] clocksource: Switched to clocksource kvm-clock [ 0.932116] VFS: Disk quotas dquot_6.6.0 [ 0.936493] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.940448] *** VALIDATE ramfs *** [ 0.942412] *** VALIDATE hugetlbfs *** [ 0.944703] pnp: PnP ACPI init [ 0.947733] pnp: PnP ACPI: found 6 devices [ 0.967169] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.971156] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.974249] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.976866] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.979904] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.982726] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.985748] NET: Registered protocol family 2 [ 0.988793] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.994228] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.999417] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.005711] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.010277] TCP: Hash tables configured (established 65536 bind 65536) [ 1.014275] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.018267] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.021414] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.024809] NET: Registered protocol family 1 [ 1.028752] RPC: Registered named UNIX socket transport module. [ 1.031263] RPC: Registered udp transport module. [ 1.033381] RPC: Registered tcp transport module. [ 1.035555] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.038197] NET: Registered protocol family 44 [ 1.040271] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.042776] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.045135] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.047396] PCI: CLS 0 bytes, default 64 [ 1.049144] Unpacking initramfs... [ 2.679408] debug: unmapping init [mem 0xffff9e61bcc54000-0xffff9e61bffbffff] [ 2.695278] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.701891] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.711109] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.221025] hrtimer: interrupt took 5012654 ns [ 3.808464] Initialise system trusted keyrings [ 3.810438] Key type blacklist registered [ 3.814722] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.838892] zbud: loaded [ 3.849622] *** VALIDATE nfs *** [ 3.855760] *** VALIDATE nfs4 *** [ 3.861984] pstore: using deflate compression [ 3.876562] Platform Keyring initialized [ 4.246629] NET: Registered protocol family 38 [ 4.249240] Key type asymmetric registered [ 4.254154] Asymmetric key parser 'x509' registered [ 4.257573] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.262568] io scheduler mq-deadline registered [ 4.264839] io scheduler kyber registered [ 4.270184] io scheduler bfq registered [ 4.278828] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.286443] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.297295] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.307964] ACPI: Power Button [PWRF] [ 4.347369] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.373981] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.409564] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.441542] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.498666] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.547966] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.599650] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.610514] Non-volatile memory driver v1.3 [ 4.614403] Linux agpgart interface v0.103 [ 4.719759] virtio_blk virtio1: [vda] 150608 512-byte logical blocks (77.1 MB/73.5 MiB) [ 4.730893] vda: detected capacity change from 0 to 77111296 [ 4.758806] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.764617] vdb: detected capacity change from 0 to 1073741824 [ 4.807859] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.811850] vdc: detected capacity change from 0 to 2621440000 [ 4.856200] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.872692] vdd: detected capacity change from 0 to 2621440000 [ 4.909779] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.921982] vde: detected capacity change from 0 to 4294967296 [ 4.954518] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.961445] vdf: detected capacity change from 0 to 4294967296 [ 4.995422] libphy: Fixed MDIO Bus: probed [ 5.004958] usbcore: registered new interface driver usbserial_generic [ 5.009473] usbserial: USB Serial support registered for generic [ 5.014382] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.022272] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.026682] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.037910] mousedev: PS/2 mouse device common for all mice [ 5.043242] rtc_cmos 00:05: RTC can wake from S4 [ 5.055688] rtc_cmos 00:05: registered as rtc0 [ 5.058810] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.065383] intel_pstate: CPU model not supported [ 5.069196] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.071749] hid: raw HID events driver (C) Jiri Kosina [ 5.086157] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.092050] usbcore: registered new interface driver usbhid [ 5.092061] usbhid: USB HID core driver [ 5.092213] drop_monitor: Initializing network drop monitor service [ 5.092358] Initializing XFRM netlink socket [ 5.092738] NET: Registered protocol family 10 [ 5.100419] Segment Routing with IPv6 [ 5.115311] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.116465] NET: Registered protocol family 17 [ 5.146348] mpls_gso: MPLS GSO support [ 5.152371] RAS: Correctable Errors collector initialized. [ 5.154632] AVX version of gcm_enc/dec engaged. [ 5.157517] AES CTR mode by8 optimization enabled [ 5.382597] sched_clock: Marking stable (5382574188, 0)->(6291937724, -909363536) [ 5.394228] registered taskstats version 1 [ 5.397417] Loading compiled-in X.509 certificates [ 5.402354] zswap: loaded using pool lzo/zbud [ 5.484708] Key type big_key registered [ 5.517492] Key type encrypted registered [ 5.520575] ima: No TPM chip found, activating TPM-bypass! [ 5.526095] ima: Allocated hash algorithm: sha1 [ 5.530100] ima: No architecture policies found [ 5.533225] evm: Initialising EVM extended attributes: [ 5.534862] evm: security.selinux [ 5.538469] evm: security.ima [ 5.541151] evm: security.capability [ 5.544542] evm: HMAC attrs: 0x1 [ 5.548810] rtc_cmos 00:05: setting system clock to 2026-09-29 19:35:43 UTC (1790710543) [ 5.562437] debug: unmapping init [mem 0xffffffffa7603000-0xffffffffa77fffff] [ 5.570520] debug: unmapping init [mem 0xffffffffa6382000-0xffffffffa6658fff] [ 5.590106] Write protecting the kernel read-only data: 28672k [ 5.596575] debug: unmapping init [mem 0xffffffffa4a03000-0xffffffffa4bfffff] [ 5.603661] debug: unmapping init [mem 0xffffffffa5314000-0xffffffffa53fffff] [ 5.693794] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.712097] systemd[1]: Detected virtualization kvm. [ 5.714270] systemd[1]: Detected architecture x86-64. [ 5.718160] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.761877] systemd[1]: No hostname configured. [ 5.768323] systemd[1]: Set hostname to . [ 5.771213] random: systemd: uninitialized urandom read (16 bytes read) [ 5.775248] systemd[1]: Initializing machine ID from random generator. [ 5.943473] random: ln: uninitialized urandom read (6 bytes read) [ 6.088089] random: systemd: uninitialized urandom read (16 bytes read) [ 6.090585] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 6.101768] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 6.110623] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.755829] device-mapper: uevent: version 1.0.3 [ 7.759651] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.614149] random: fast init done [ 9.639499] virtio_net virtio0 ens2: renamed from eth0 [ 9.805425] scsi host0: ata_piix [ 9.846333] scsi host1: ata_piix [ 9.852194] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.857737] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.057961] random: crng init done [ 15.061643] random: 7 urandom warning(s) missed due to ratelimiting [ 15.989087] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 17.488806] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.957387] printk: systemd: 25 output lines suppressed due to ratelimiting [ 21.753398] SELinux: Disabled at runtime. [ 21.873525] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.899282] systemd[1]: Detected virtualization kvm. [ 21.902938] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.216298] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.239149] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.260365] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.270032] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.282856] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.300209] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.319989] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ 23.606514] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 24.624519] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.756305] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 25.787019] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.322973] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.401527] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 31.261618] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 32.120592] NFS: Registering the id_resolver key type [ 32.123305] Key type id_resolver registered [ 32.125603] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg426-server login: [ 104.318626] libcfs: loading out-of-tree module taints kernel. [ 104.440547] Key type ._llcrypt registered [ 104.442886] Key type .llcrypt registered [ 104.590077] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 122.807188] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 124.169525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 124.187469] alg: No test for adler32 (adler32-zlib) [ 125.576163] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 126.445907] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 128.161909] Key type lgssc registered [ 129.769381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 150.867470] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 191.902534] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 206.283679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 206.324718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 207.653590] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 207.686230] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 207.813987] Lustre: lustre-MDT0000: new disk, initializing [ 207.963442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 208.013073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 213.512186] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 227.498315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 227.610404] Lustre: 6509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 227.637601] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 227.641390] Lustre: Skipped 1 previous similar message [ 227.718241] Lustre: lustre-MDT0001: new disk, initializing [ 227.799074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 227.863652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 227.873706] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 232.124352] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.119184] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 246.399777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 246.683139] Lustre: lustre-OST0000: new disk, initializing [ 246.687134] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 246.695700] Lustre: 8448:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 246.800903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 248.223363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 248.234727] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 248.341209] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 252.203409] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 266.164070] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 266.318909] Lustre: lustre-OST0001: new disk, initializing [ 266.324373] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 266.330114] Lustre: 9520:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 266.400642] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 272.112494] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 273.935714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 273.946031] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 274.007565] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 282.687386] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 289.045818] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 294.921284] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing check_logdir /tmp/testlogs/ [ 299.585091] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing yml_node [ 303.524404] Lustre: DEBUG MARKER: Client: 2.17.59.2 [ 305.635806] Lustre: DEBUG MARKER: MDS: 2.17.59.2 [ 308.224414] Lustre: DEBUG MARKER: OSS: 2.17.59.2 [ 309.605822] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 29 15:40:47 EDT 2026 [ 326.037489] Lustre: DEBUG MARKER: only running tests: 33-75 [ 327.326412] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 328.682526] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 345.568947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 345.570927] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 345.582170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 345.604903] Lustre: Skipped 3 previous similar messages [ 349.935528] Lustre: server umount lustre-MDT0000 complete [ 353.737374] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790710891 with bad export cookie 7810596041835309999 [ 353.741168] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 353.754534] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 355.810439] LustreError: 9529:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 355.811560] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 355.811871] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 355.811875] Lustre: Skipped 3 previous similar messages [ 355.827839] LustreError: 9529:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 355.845670] Lustre: Skipped 1 previous similar message [ 358.816722] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 358.817599] LustreError: 6516:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 358.822734] Lustre: Skipped 1 previous similar message [ 358.860731] LustreError: 6516:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 363.939997] LustreError: 6515:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 363.940518] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 363.958206] Lustre: Skipped 2 previous similar messages [ 368.097810] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 368.312446] Lustre: server umount lustre-MDT0001 complete [ 372.520764] Lustre: server umount lustre-OST0000 complete [ 376.122707] Lustre: server umount lustre-OST0001 complete [ 383.037822] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 392.649205] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 433.790423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 434.038484] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 434.072814] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 434.175818] Lustre: lustre-MDT0000: new disk, initializing [ 434.238442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 434.258323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 438.795265] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 449.805292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 449.948871] Lustre: 15124:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 450.046222] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 450.055672] Lustre: Skipped 1 previous similar message [ 450.193898] Lustre: lustre-MDT0001: new disk, initializing [ 450.319793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 450.361954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 450.381072] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 454.925454] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 464.318213] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 470.255250] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 477.316128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 477.696834] Lustre: lustre-OST0000: new disk, initializing [ 477.699974] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 477.717330] Lustre: 17093:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 477.808080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 479.428507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 479.438972] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 479.582496] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 483.352067] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 492.540531] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 498.176875] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 498.559741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 505.833940] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 506.159835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 510.448808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.452978] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 510.467991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 510.482901] Lustre: Skipped 1 previous similar message [ 510.523165] LustreError: Skipped 1 previous similar message [ 520.688573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 520.710501] Lustre: Skipped 4 previous similar messages [ 523.232034] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 523.555976] Lustre: server umount lustre-OST0000 complete [ 530.913838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.936104] Lustre: Skipped 1 previous similar message [ 534.741599] Lustre: server umount lustre-MDT0000 complete [ 536.032924] LustreError: 15130:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.050400] LustreError: 15130:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 539.130541] LustreError: 17091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790711077 with bad export cookie 7810596041835311532 [ 539.135523] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 539.154364] LustreError: 17091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 539.828586] Lustre: server umount lustre-MDT0001 complete [ 545.042563] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 15:44:42 (1790711082) [ 547.498907] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 549.420556] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 15:44:46 (1790711086) [ 557.062527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.427102] LustreError: 19502:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.491962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 562.363954] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 567.776170] LustreError: 19502:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.796911] LustreError: 19502:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 571.421397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 576.428471] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 583.094180] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 589.409431] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 596.994343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 604.207108] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 611.659514] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 622.051921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 622.061258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 622.073261] Lustre: Skipped 1 previous similar message [ 622.078718] Lustre: Skipped 2 previous similar messages [ 623.773537] Lustre: server umount lustre-OST0000 complete [ 628.197170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 632.291680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 632.310527] Lustre: Skipped 1 previous similar message [ 633.722650] Lustre: server umount lustre-MDT0000 complete [ 637.266778] LustreError: 19483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790711175 with bad export cookie 7810596041835312407 [ 637.274263] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 637.279886] LustreError: 19483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 637.587388] Lustre: server umount lustre-MDT0001 complete [ 648.347344] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 651.283865] Key type lgssc unregistered [ 651.646656] LNet: 23035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 651.661432] LNetError: 23035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 651.680847] LNet: Removed LNI 192.168.204.126@tcp [ 652.647344] Key type .llcrypt unregistered [ 652.651655] Key type ._llcrypt unregistered [ 674.631681] Key type ._llcrypt registered [ 674.634650] Key type .llcrypt registered [ 674.748083] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 675.866451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 675.881312] alg: No test for adler32 (adler32-zlib) [ 676.955423] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 677.204566] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 678.952262] Key type lgssc registered [ 680.347113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 689.511441] loop: module loaded [ 689.611959] loop0: detected capacity change from 0 to 204800000 [ 689.879915] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 689.901979] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 690.410207] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 697.652652] loop0: detected capacity change from 0 to 204800000 [ 697.823020] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 697.840795] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 697.867640] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 704.035303] loop0: detected capacity change from 0 to 204800000 [ 709.815876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 709.858609] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 711.139115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 711.207064] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 711.321354] Lustre: lustre-MDT0000: new disk, initializing [ 711.433333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 711.455217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 716.283166] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 725.835427] loop1: detected capacity change from 0 to 204800000 [ 732.004294] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 732.296394] Lustre: lustre-OST07c6: new disk, initializing [ 732.300239] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 732.307669] Lustre: Skipped 1 previous similar message [ 732.310989] Lustre: 26244:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 732.404170] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 739.671355] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 741.574829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 741.585670] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 741.638951] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 745.979385] Lustre: 25229:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 746.004936] Lustre: 25229:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 746.015841] Lustre: 25229:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 746.027517] Lustre: 25229:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 746.043843] Lustre: 25229:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 746.055452] Lustre: 25229:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 763.505352] Lustre: 25228:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 763.514725] Lustre: 25228:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 763.520695] Lustre: 25228:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 763.530470] Lustre: 25228:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 763.537893] Lustre: 25228:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 763.551900] Lustre: 25228:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 768.440568] Lustre: 25229:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 768.448767] Lustre: 25229:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 768.454611] Lustre: 25229:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 768.459625] Lustre: 25229:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.467181] Lustre: 25229:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 768.472620] Lustre: 25229:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.478036] Lustre: 25229:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 768.488804] Lustre: 25229:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.499774] Lustre: 25229:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 768.509870] Lustre: 25229:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.517704] Lustre: 25229:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 768.523734] Lustre: 25229:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.713256] Lustre: 25230:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 771.723050] Lustre: 25230:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 771.743213] Lustre: 25230:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 771.751455] Lustre: 25230:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.759475] Lustre: 25230:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 771.765590] Lustre: 25230:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.772088] Lustre: 25230:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 771.778941] Lustre: 25230:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.786235] Lustre: 25230:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 771.792452] Lustre: 25230:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.798087] Lustre: 25230:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 771.803753] Lustre: 25230:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.280449] Lustre: 25228:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 776.291833] Lustre: 25228:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 776.302744] Lustre: 25228:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 776.313577] Lustre: 25228:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.320151] Lustre: 25228:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 776.330758] Lustre: 25228:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.337031] Lustre: 25228:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 776.345784] Lustre: 25228:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.351360] Lustre: 25228:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 776.362720] Lustre: 25228:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.371164] Lustre: 25228:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 776.381019] Lustre: 25228:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 787.177860] Lustre: 25230:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 787.189705] Lustre: 25230:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 5 previous similar messages [ 787.198154] Lustre: 25230:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 787.208464] Lustre: 25230:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 787.213149] Lustre: 25230:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 787.221101] Lustre: 25230:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 787.229534] Lustre: 25230:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 787.239453] Lustre: 25230:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 787.247846] Lustre: 25230:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 787.257274] Lustre: 25230:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 787.264016] Lustre: 25230:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 787.268637] Lustre: 25230:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 839.138914] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 839.149602] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 844.258397] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 848.351125] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 848.554659] Lustre: server umount lustre-OST07c6 complete [ 858.540938] Lustre: server umount lustre-MDT0000 complete [ 875.721543] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 878.548175] Key type lgssc unregistered [ 878.896682] LNet: 28800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.911314] LNetError: 28800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.976294] LNet: Removed LNI 192.168.204.126@tcp [ 880.950176] Key type .llcrypt unregistered [ 880.957635] Key type ._llcrypt unregistered [ 902.971267] Key type ._llcrypt registered [ 902.977140] Key type .llcrypt registered [ 903.105400] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 904.021158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 904.043647] alg: No test for adler32 (adler32-zlib) [ 905.021637] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 905.339174] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 907.047242] Key type lgssc registered [ 908.290839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 917.211823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 917.243530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 918.713745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 923.317718] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 928.226599] LustreError: 30088:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 928.246479] LustreError: 30088:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 932.223392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 932.669913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 937.428555] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 945.887624] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 952.282352] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 960.275690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 960.700940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 963.841483] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 968.184436] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 977.366511] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 994.784719] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 994.793104] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 994.793703] LustreError: Skipped 1 previous similar message [ 994.814034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 995.704253] Lustre: server umount lustre-OST0000 complete [ 1002.981262] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1002.988131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1003.001231] Lustre: Skipped 1 previous similar message [ 1003.019148] Lustre: Skipped 1 previous similar message [ 1005.728692] Lustre: server umount lustre-MDT0000 complete [ 1008.101943] LustreError: 30089:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1008.126179] LustreError: 30089:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1009.712707] LustreError: 30067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790711547 with bad export cookie 14564146166748901109 [ 1009.730230] LustreError: 30067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1009.730241] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1020.066485] Lustre: server umount lustre-MDT0001 complete [ 1032.150992] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1035.176920] Key type lgssc unregistered [ 1035.561404] LNet: 33822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.582235] LNetError: 33822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1035.602797] LNet: Removed LNI 192.168.204.126@tcp [ 1036.504927] Key type .llcrypt unregistered [ 1036.507945] Key type ._llcrypt unregistered [ 1053.343420] Key type ._llcrypt registered [ 1053.345806] Key type .llcrypt registered [ 1053.475438] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1054.478332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1054.508619] alg: No test for adler32 (adler32-zlib) [ 1055.533223] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 1055.765088] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1057.431270] Key type lgssc registered [ 1058.405027] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1063.986760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1064.003207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1065.335834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1068.736944] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1075.681995] LustreError: 35087:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1075.704743] LustreError: 35087:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1076.678387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1077.029639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1081.385321] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1089.196435] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1095.116962] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1101.820694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1102.182151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1107.708158] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1110.428437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1115.749572] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1132.006172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1132.018665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1132.027369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1132.040403] Lustre: Skipped 1 previous similar message [ 1132.057578] Lustre: Skipped 1 previous similar message [ 1137.131996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1137.136382] Lustre: Skipped 1 previous similar message [ 1142.246536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1142.263934] Lustre: Skipped 1 previous similar message [ 1145.825600] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1145.975558] Lustre: server umount lustre-OST0000 complete [ 1152.485693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1152.501537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1152.516901] Lustre: Skipped 1 previous similar message [ 1157.602972] LustreError: 35086:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1157.644158] LustreError: 35086:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1157.720252] Lustre: server umount lustre-MDT0000 complete [ 1161.809125] LustreError: 35067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790711699 with bad export cookie 11549010500959038713 [ 1161.818041] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1161.825458] LustreError: 35067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1162.174957] Lustre: server umount lustre-MDT0001 complete [ 1175.298989] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1177.864729] Key type lgssc unregistered [ 1178.183562] LNet: 38643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1178.199741] LNetError: 38643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1178.216741] LNet: Removed LNI 192.168.204.126@tcp [ 1178.924834] Key type .llcrypt unregistered [ 1178.929568] Key type ._llcrypt unregistered [ 1211.504798] Key type ._llcrypt registered [ 1211.506750] Key type .llcrypt registered [ 1211.703504] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1213.146453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1213.192109] alg: No test for adler32 (adler32-zlib) [ 1214.328609] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 1214.712322] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1216.431347] Key type lgssc registered [ 1217.924446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1227.054459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1227.068554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1228.511985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1232.743118] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1239.012555] LustreError: 40086:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1239.032632] LustreError: 40086:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1243.248892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1243.638585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1248.514110] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1255.557338] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1263.377708] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1271.060855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1271.462532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1273.518000] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1278.252074] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1287.024586] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1294.307972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1294.310831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1294.325871] Lustre: Skipped 2 previous similar messages [ 1294.349064] Lustre: Skipped 2 previous similar messages [ 1295.223697] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 1299.431359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1299.450349] Lustre: Skipped 2 previous similar messages [ 1300.656840] Lustre: server umount lustre-MDT0000 complete [ 1304.547309] LustreError: 40082:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1304.573974] LustreError: 40082:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1305.016175] LustreError: 40067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790711842 with bad export cookie 4365411771049742070 [ 1305.030170] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1305.034225] LustreError: 40067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1305.462782] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [ 1305.482946] Lustre: Skipped 1 previous similar message [ 1309.665381] LustreError: 42841:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1309.675402] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1309.684981] LustreError: 42841:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1309.701330] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1315.655669] LustreError: 42841:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1315.675462] LustreError: 42841:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1319.391192] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1319.607173] Lustre: server umount lustre-MDT0001 complete [ 1335.081874] Lustre: server umount lustre-OST0000 complete [ 1353.655478] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1356.956356] Key type lgssc unregistered [ 1357.317208] LNet: 43945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1357.326224] LNetError: 43945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1357.349530] LNet: Removed LNI 192.168.204.126@tcp [ 1358.331815] Key type .llcrypt unregistered [ 1358.333807] Key type ._llcrypt unregistered [ 1385.384318] Key type ._llcrypt registered [ 1385.390193] Key type .llcrypt registered [ 1385.561818] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1386.515333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1386.557751] alg: No test for adler32 (adler32-zlib) [ 1387.609804] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 1387.880937] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1389.551244] Key type lgssc registered [ 1391.051389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1399.624382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1399.640754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1401.173558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1406.162317] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1411.565402] LustreError: 45375:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1411.587513] LustreError: 45375:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1415.283059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1415.678874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1421.234974] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1430.094698] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1437.706263] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1445.891514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1446.276780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1447.353480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1453.502314] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1462.011162] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1467.872928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1467.892242] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1467.906090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1468.795213] Lustre: lustre-OST0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 1470.948524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1470.950915] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1470.954900] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1470.966656] Lustre: Skipped 1 previous similar message [ 1473.885283] Lustre: lustre-OST0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 1474.151655] Lustre: server umount lustre-OST0000 complete [ 1496.547135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1496.573567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1496.593966] Lustre: Skipped 1 previous similar message [ 1496.626450] Lustre: Skipped 1 previous similar message [ 1500.654945] Lustre: server umount lustre-MDT0000 complete [ 1501.675649] LustreError: 45375:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1501.707552] LustreError: 45375:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1505.234159] LustreError: 45357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790712043 with bad export cookie 6205383616807214261 [ 1505.237034] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1505.244034] LustreError: 45357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1511.514910] Lustre: server umount lustre-MDT0001 complete [ 1526.656792] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1529.988188] Key type lgssc unregistered [ 1530.221791] LNet: 49100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1530.261769] LNetError: 49100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1530.277304] LNet: Removed LNI 192.168.204.126@tcp [ 1531.730499] Key type .llcrypt unregistered [ 1531.732386] Key type ._llcrypt unregistered [ 1560.508703] Key type ._llcrypt registered [ 1560.512391] Key type .llcrypt registered [ 1560.707774] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1561.926042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1561.948448] alg: No test for adler32 (adler32-zlib) [ 1562.965886] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 1563.325374] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1565.040101] Key type lgssc registered [ 1566.589437] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1575.468672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1575.506550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1576.988121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1581.968052] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1586.150589] LustreError: 50536:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1586.174710] LustreError: 50536:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1591.269405] LustreError: 50535:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1591.354053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1591.728143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1596.692865] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1605.426923] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1612.502300] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1620.411039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1620.874107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1621.894138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1628.157368] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1636.921398] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1643.544865] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1647.150418] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1659.657901] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-2916h03m16s [ 1661.550867] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1665.396519] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 1666.441652] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 1667.041767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1667.061149] Lustre: Skipped 2 previous similar messages [ 1668.146019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1668.155553] Lustre: Skipped 5 previous similar messages [ 1671.500698] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 1677.648490] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 1677.657848] Lustre: Skipped 3 previous similar messages [ 1678.305909] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1678.448540] Lustre: server umount lustre-MDT0000 complete [ 1680.355404] LustreError: 50535:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1680.382406] LustreError: 50535:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1683.375976] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1683.792099] LustreError: 50532:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1689.928566] LustreError: 50531:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1690.005222] LustreError: 50531:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1692.379302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1692.578239] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1692.980214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1693.056964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1693.999800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1697.598737] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1700.982485] Lustre: DEBUG MARKER: Wait for df (36910) ... [ 1703.115899] Lustre: DEBUG MARKER: done [ 1708.000466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1708.022878] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1708.048782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1708.513910] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1722.335139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1722.518584] Lustre: server umount lustre-OST0000 complete [ 1730.018128] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1730.025985] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1730.034561] Lustre: Skipped 1 previous similar message [ 1730.040587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1730.057327] Lustre: Skipped 8 previous similar messages [ 1733.815587] Lustre: server umount lustre-MDT0000 complete [ 1737.198691] LustreError: 52432:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1737.210628] LustreError: 52432:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1738.102749] LustreError: 50515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790712276 with bad export cookie 8671999884599814473 [ 1738.122179] LustreError: 50515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1738.123696] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1738.527224] Lustre: server umount lustre-MDT0001 complete [ 1751.901634] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1754.928094] Key type lgssc unregistered [ 1755.326032] LNet: 55459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1755.341049] LNetError: 55459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1755.362351] LNet: Removed LNI 192.168.204.126@tcp [ 1756.480282] Key type .llcrypt unregistered [ 1756.486666] Key type ._llcrypt unregistered [ 1800.428771] Key type ._llcrypt registered [ 1800.431419] Key type .llcrypt registered [ 1800.577367] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1801.925282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1801.963334] alg: No test for adler32 (adler32-zlib) [ 1803.180399] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 1803.414854] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1805.087227] Key type lgssc registered [ 1806.397990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1814.813702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1814.843982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1816.152649] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1816.172704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1816.350183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1821.021690] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1831.531761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1831.642040] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1831.685308] Lustre: 57681:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1832.086576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1837.295597] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1848.047493] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1857.142832] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1865.868964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1866.147906] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1866.399174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1872.056540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1873.866783] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1881.865900] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1890.321919] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-2916h07m07s [ 1892.618725] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1901.448530] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1904.154204] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1904.314661] LustreError: 57691:0:(mdt_handler.c:2345:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1924.492355] Lustre: lustre-MDT0000: Client 0f1d20a5-a98c-4373-8eb7-ae074f101c43 (at 192.168.204.26@tcp) reconnecting [ 1944.384809] LustreError: 57691:0:(mdt_handler.c:2345:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1947.212055] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1950.383545] Lustre: DEBUG MARKER: done [ 1958.883137] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1958.894543] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1958.914421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1959.393852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1959.414332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1959.434906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1959.444951] Lustre: Skipped 1 previous similar message [ 1963.325809] Lustre: server umount lustre-OST0000 complete [ 1969.635076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1969.647126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1969.647166] Lustre: Skipped 1 previous similar message [ 1974.118771] Lustre: server umount lustre-MDT0000 complete [ 1978.873149] LustreError: 57675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790712516 with bad export cookie 13838044511941001119 [ 1978.874404] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1978.880051] LustreError: 57675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1979.259448] Lustre: server umount lustre-MDT0001 complete [ 1995.228426] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1998.743033] Key type lgssc unregistered [ 1999.251559] LNet: 62019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1999.266937] LNetError: 62019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1999.287775] LNet: Removed LNI 192.168.204.126@tcp [ 2000.447341] Key type .llcrypt unregistered [ 2000.450465] Key type ._llcrypt unregistered [ 2052.547625] Key type ._llcrypt registered [ 2052.556296] Key type .llcrypt registered [ 2052.696711] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2054.217250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2054.249325] alg: No test for adler32 (adler32-zlib) [ 2055.363237] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 2055.607669] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2057.359368] Key type lgssc registered [ 2058.799794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2062.510656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2062.525330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2063.765839] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2063.781801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2063.934732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2064.891346] Lustre: Failing over lustre-MDT0000 [ 2065.233349] Lustre: server umount lustre-MDT0000 complete [ 2071.845694] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 16:10:09 (1790712609) [ 2078.726392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2079.164404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2084.059673] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2094.072137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2094.264557] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2094.310324] Lustre: 64834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2094.563909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2100.120796] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2108.534022] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2115.977456] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2123.359514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2123.646121] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2123.909077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2131.432535] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2132.093225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2140.104516] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2146.617274] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2156.003079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2156.020569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2156.035884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2158.050119] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2158.058668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2158.078208] Lustre: Skipped 2 previous similar messages [ 2159.593620] Lustre: server umount lustre-MDT0000 complete [ 2163.798119] LustreError: 64826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790712701 with bad export cookie 16646241862911126821 [ 2163.816462] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2163.816713] LustreError: 64826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2164.260576] Lustre: server umount lustre-MDT0001 complete [ 2168.091471] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2173.273567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2183.868696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2184.672745] Lustre: 63746:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790712706/real 1790712706] req@00000000d733a107 x1877698247961088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790712722 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2184.699507] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2184.711175] Lustre: Skipped 1 previous similar message [ 2189.286407] LustreError: 66684:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2189.312111] LustreError: 66684:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2189.417505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2189.517918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2190.369113] Lustre: 63744:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790712711/real 1790712711] req@000000004ddafcb3 x1877698247961216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790712727 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2193.451433] LustreError: 66685:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2194.464832] Lustre: 63745:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790712716/real 1790712716] req@000000000e694fb4 x1877698247961472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790712732 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2194.492665] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2194.534079] LustreError: 66684:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2194.584394] LustreError: 66684:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2199.663646] Lustre: 63747:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790712721/real 1790712721] req@000000004a8e7ae5 x1877698247961600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790712737 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2200.047583] LustreError: 68608:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2200.072566] LustreError: 68608:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2205.162678] LustreError: 66685:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2205.193893] LustreError: 66685:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2205.473893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2205.901445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2211.327024] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2212.280400] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2213.435503] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2221.130915] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2226.802929] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2233.832119] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2233.838637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2233.841951] Lustre: Skipped 1 previous similar message [ 2233.863493] Lustre: Skipped 2 previous similar messages [ 2234.858630] Lustre: server umount lustre-MDT0000 complete [ 2238.950688] LustreError: 68606:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.968590] LustreError: 68606:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2239.039287] LustreError: 64825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790712776 with bad export cookie 16646241862911129096 [ 2239.064722] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2239.069093] LustreError: 64825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2239.406551] Lustre: server umount lustre-MDT0001 complete [ 2246.263132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2256.759936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.447649] Lustre: 63744:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790712782/real 1790712782] req@0000000041776812 x1877698247989120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790712798 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2260.492575] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2260.505697] Lustre: Skipped 1 previous similar message [ 2264.615466] LustreError: 63743:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009b853561 x1877698247989888/t0(0) o250->MGC192.168.204.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2264.943262] LustreError: 66684:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2265.038261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2265.093173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2269.789770] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2270.482666] Lustre: 63747:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790712792/real 1790712792] req@00000000e2b17a61 x1877698247989504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790712808 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2270.516417] Lustre: 63747:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2279.336215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2285.050321] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2285.162637] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2289.199050] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2294.088067] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2299.841343] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2309.607379] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2309.610775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2309.627948] Lustre: Skipped 1 previous similar message [ 2309.644968] Lustre: Skipped 2 previous similar messages [ 2312.457277] Lustre: server umount lustre-MDT0000 complete [ 2314.722852] LustreError: 71286:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2314.759959] LustreError: 71286:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 2316.800637] LustreError: 65569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790712854 with bad export cookie 16646241862911130475 [ 2316.805684] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2316.819134] LustreError: 65569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2317.383994] Lustre: server umount lustre-MDT0001 complete [ 2324.863966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2328.545992] Lustre: DEBUG MARKER: files compared the same [ 2340.986644] Lustre: server umount lustre-OST0000 complete [ 2358.931719] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2361.585995] Key type lgssc unregistered [ 2361.908626] LNet: 74564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2361.919715] LNetError: 74564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2361.946309] LNet: Removed LNI 192.168.204.126@tcp [ 2362.866521] Key type .llcrypt unregistered [ 2362.868263] Key type ._llcrypt unregistered [ 2389.695453] Key type ._llcrypt registered [ 2389.697811] Key type .llcrypt registered [ 2389.820561] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2390.818516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2390.857628] alg: No test for adler32 (adler32-zlib) [ 2392.096546] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 2392.429388] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2394.111346] Key type lgssc registered [ 2395.418292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2403.846677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2403.876351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2405.414629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2410.623961] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug +malloc all [ 2415.585884] LustreError: 75998:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.604652] LustreError: 75998:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2420.558963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2420.703924] LustreError: 75997:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2421.053511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2425.471893] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug +malloc all [ 2433.124359] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2439.321887] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2446.018276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2446.377764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2450.480111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2452.998473] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug +malloc all [ 2460.697109] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2475.999929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2476.004108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2476.004565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2476.014335] LustreError: Skipped 1 previous similar message [ 2476.050793] Lustre: Skipped 1 previous similar message [ 2479.660765] Lustre: server umount lustre-OST0000 complete [ 2486.240617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2486.254553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2486.262463] Lustre: Skipped 2 previous similar messages [ 2489.606989] Lustre: server umount lustre-MDT0000 complete [ 2491.362585] LustreError: 75998:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2491.385421] LustreError: 75998:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2494.109119] LustreError: 75979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790713032 with bad export cookie 15377016195565724875 [ 2494.118183] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2494.120386] LustreError: 75979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2494.674451] Lustre: server umount lustre-MDT0001 complete [ 2507.384527] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2510.541971] Key type lgssc unregistered [ 2510.896685] LNet: 79551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2510.907525] LNetError: 79551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2510.920111] LNet: Removed LNI 192.168.204.126@tcp [ 2511.737253] Key type .llcrypt unregistered [ 2511.739599] Key type ._llcrypt unregistered [ 2539.555597] Key type ._llcrypt registered [ 2539.558136] Key type .llcrypt registered [ 2539.691199] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2540.920268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2540.975548] alg: No test for adler32 (adler32-zlib) [ 2542.375857] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 2542.718815] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2544.503726] Key type lgssc registered [ 2545.980629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2554.506488] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2554.524891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2570.143494] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2575.263521] LustreError: 80956:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.126@tcp: failed processing log, type 4: rc = -110 [ 2601.953387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2609.534511] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2618.423291] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2626.634123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2626.732149] Lustre: *** cfs_fail_loc=706, val=0*** [ 2627.317782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2627.441075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2632.241966] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2632.684514] LustreError: 80981:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2632.711680] LustreError: 80981:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2636.771739] LustreError: 80982:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2636.812154] LustreError: 80982:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2641.909480] LustreError: 82041:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.950287] LustreError: 82041:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2642.770535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2643.293274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2648.479819] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2657.728544] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2665.354630] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2668.513079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2668.523404] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2668.553918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2669.054381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2669.060208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2669.083739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2669.108215] Lustre: Skipped 1 previous similar message [ 2674.155554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2674.169986] Lustre: Skipped 1 previous similar message [ 2679.275479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2682.335150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2682.558974] Lustre: server umount lustre-OST0000 complete [ 2689.505230] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2689.518933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2689.525766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2689.535042] Lustre: Skipped 2 previous similar messages [ 2699.745583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2699.749602] Lustre: Skipped 3 previous similar messages [ 2702.303354] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2702.536696] Lustre: server umount lustre-MDT0000 complete [ 2704.866923] LustreError: 82046:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.885848] LustreError: 82046:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2707.897784] LustreError: 80963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790713245 with bad export cookie 13818696236854113582 [ 2707.932317] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2708.377866] Lustre: server umount lustre-MDT0001 complete [ 2721.555612] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2724.623950] Key type lgssc unregistered [ 2724.909682] LNet: 84574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2724.921533] LNetError: 84574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2724.951803] LNet: Removed LNI 192.168.204.126@tcp [ 2725.894818] Key type .llcrypt unregistered [ 2725.899111] Key type ._llcrypt unregistered [ 2755.181997] Key type ._llcrypt registered [ 2755.186300] Key type .llcrypt registered [ 2755.363944] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2757.116871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2757.140448] alg: No test for adler32 (adler32-zlib) [ 2758.335401] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 2758.732994] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2760.511981] Key type lgssc registered [ 2761.979172] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2770.672241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2770.692635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2776.757388] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2785.480483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2786.014212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2790.774094] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2791.404609] LustreError: 86697:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2791.438417] LustreError: 86697:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2795.488534] LustreError: 86696:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2795.524601] LustreError: 86696:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2800.626069] LustreError: 86700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2800.664657] LustreError: 86700:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2800.742196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2801.225458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2806.250324] LustreError: 86697:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2806.278844] LustreError: 86697:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2807.651832] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2810.340821] LustreError: 86701:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2810.360092] LustreError: 86701:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2817.373278] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2817.383993] Lustre: MGS: Client d2310514-ca6e-472d-aafb-cb9dd0bef5c7 (at 0@lo) reconnecting [ 2817.414570] LustreError: 86711:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2817.628867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2817.720113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2822.674126] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2837.991221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2837.995142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2837.999437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2838.007119] Lustre: Skipped 1 previous similar message [ 2841.569667] Lustre: server umount lustre-OST0000 complete [ 2848.225883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2848.227031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2848.239061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2848.260097] Lustre: Skipped 2 previous similar messages [ 2852.013843] Lustre: server umount lustre-MDT0000 complete [ 2853.348046] LustreError: 86701:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2853.377683] LustreError: 86701:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2856.861162] Lustre: server umount lustre-MDT0001 complete [ 2861.727808] Lustre: server umount MGS complete [ 2874.236295] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 16:23:31 (1790713411) [ 2903.141556] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 2912.605956] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2954.190825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2958.095633] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2968.872989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2969.096395] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2969.141708] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2969.261488] Lustre: lustre-MDT0001: new disk, initializing [ 2969.374140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2969.396943] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2969.405981] Lustre: 93866:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2969.420111] LustreError: 93866:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2974.191870] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2984.838336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2985.113864] Lustre: lustre-OST0000: new disk, initializing [ 2985.117810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2985.128780] Lustre: 94631:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2985.194243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2990.536458] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2993.185592] Lustre: 95087:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000b51e0c7d x1877698985107712/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790713547 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2998.820105] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3005.746421] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3005.759799] Lustre: MGS: Client 63ea58da-a8a9-4300-b7c0-23b230e1da37 (at 0@lo) reconnecting [ 3005.813031] Lustre: 93134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3005.850347] LustreError: 93852:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3005.863083] LustreError: 93852:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3005.885516] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3005.967674] Lustre: lustre-MDT0000: new disk, initializing [ 3006.135722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3006.166449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3006.267420] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3007.199991] Lustre: 95700:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000a02d2f8d x1877698985114240/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1790713561 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3010.579458] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3012.274717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3012.288878] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3012.427084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3036.651564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3036.658631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3036.674971] Lustre: Skipped 2 previous similar messages [ 3040.951918] Lustre: server umount lustre-OST0000 complete [ 3046.895345] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3046.910021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3046.910594] Lustre: Skipped 1 previous similar message [ 3046.921333] Lustre: Skipped 1 previous similar message [ 3051.383853] Lustre: server umount lustre-MDT0000 complete [ 3052.007089] LustreError: 96125:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3052.032617] LustreError: 96125:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3061.768167] Lustre: server umount lustre-MDT0001 complete [ 3065.528799] Lustre: server umount MGS complete [ 3077.178595] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 16:26:54 (1790713614) [ 3094.296886] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3096.974734] Key type lgssc unregistered [ 3097.289363] LNet: 98197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3097.300246] LNetError: 98197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3097.328916] LNet: Removed LNI 192.168.204.126@tcp [ 3098.236433] Key type .llcrypt unregistered [ 3098.246250] Key type ._llcrypt unregistered [ 3115.330322] Key type ._llcrypt registered [ 3115.332885] Key type .llcrypt registered [ 3115.443982] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3116.569780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3116.632198] alg: No test for adler32 (adler32-zlib) [ 3117.798475] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 3118.053600] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3119.816221] Key type lgssc registered [ 3121.140117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3129.933496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3129.951400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3131.516276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3136.277309] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3141.098097] LustreError: 99434:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3141.128848] LustreError: 99434:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3145.159972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3145.498614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3149.771970] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3156.212661] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3163.419850] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3172.026101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3172.431084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3176.537444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3180.910532] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3189.213505] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3200.992913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3201.000356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3201.001301] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3201.015882] Lustre: Skipped 1 previous similar message [ 3201.032517] Lustre: Skipped 1 previous similar message [ 3205.789513] Lustre: server umount lustre-OST0000 complete [ 3211.241057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3211.246680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3211.261660] Lustre: Skipped 1 previous similar message [ 3211.268972] Lustre: Skipped 1 previous similar message [ 3215.941422] Lustre: server umount lustre-MDT0000 complete [ 3216.358389] LustreError: 99435:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3216.396763] LustreError: 99435:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3220.183913] LustreError: 99414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790713758 with bad export cookie 15599228866897767067 [ 3220.191136] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3220.191885] LustreError: 99414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3220.494302] Lustre: server umount lustre-MDT0001 complete [ 3232.450725] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3235.347111] Key type lgssc unregistered [ 3235.597784] LNet: 102972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3235.603802] LNetError: 102972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3235.627226] LNet: Removed LNI 192.168.204.126@tcp [ 3236.459451] Key type .llcrypt unregistered [ 3236.463408] Key type ._llcrypt unregistered [ 3255.364567] Key type ._llcrypt registered [ 3255.366919] Key type .llcrypt registered [ 3255.507122] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3256.413646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3256.429911] alg: No test for adler32 (adler32-zlib) [ 3257.506463] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 3257.758245] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3259.455241] Key type lgssc registered [ 3261.210055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3268.398841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3268.401670] LustreError: 104074:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3268.405993] Lustre: Skipped 1 previous similar message [ 3268.422152] LustreError: 104073:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3268.426580] LustreError: 104074:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4990 [ 3268.441304] LustreError: 104074:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3268.454535] LustreError: 104074:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3268.477278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3270.182963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3278.772193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3279.055217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3284.139528] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3290.015447] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing lsmod [ 3293.169395] LustreError: 105281:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3293.187040] LustreError: 105282:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3293.194032] LustreError: 105281:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4991 [ 3293.211596] LustreError: 105281:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3293.218259] LustreError: 105281:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3293.219392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3293.609610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3295.662394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3299.813319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3299.818211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3299.839875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3300.843330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3300.844736] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3300.879802] Lustre: Skipped 1 previous similar message [ 3302.845696] Lustre: server umount lustre-MDT0000 complete [ 3303.974960] LustreError: 105002:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3304.006413] LustreError: 105002:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3307.325389] LustreError: 104090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790713845 with bad export cookie 6765105326919096550 [ 3307.340513] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3307.345449] LustreError: 104090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3308.127391] LustreError: 104543:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c3f41de4 x1877699508327296/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3308.429753] Lustre: server umount lustre-MDT0001 complete [ 3323.181719] Lustre: server umount lustre-OST0000 complete [ 3333.445262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3333.903689] LustreError: 106356:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3334.004848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3338.841703] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3339.234371] LustreError: 106357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3343.339900] LustreError: 106356:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3348.157393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3348.459093] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3348.462158] Lustre: Skipped 2 previous similar messages [ 3348.623864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3353.526734] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3361.110556] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3368.033654] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3376.051343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3376.591780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3378.681123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3382.529823] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3390.532809] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3402.735679] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3402.748572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3402.762601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3404.261211] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3407.584914] Lustre: server umount lustre-OST0000 complete [ 3414.496294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3414.515499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3414.521508] Lustre: Skipped 2 previous similar messages [ 3414.549624] Lustre: Skipped 3 previous similar messages [ 3419.618442] LustreError: 106352:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3419.638297] LustreError: 106352:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3419.788344] Lustre: server umount lustre-MDT0000 complete [ 3424.188347] LustreError: 106337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790713962 with bad export cookie 6765105326919097467 [ 3424.190354] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3424.209403] LustreError: 106337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3424.618239] Lustre: server umount lustre-MDT0001 complete [ 3436.512199] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3439.467823] Key type lgssc unregistered [ 3439.856746] LNet: 109891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3439.872496] LNetError: 109891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3439.892871] LNet: Removed LNI 192.168.204.126@tcp [ 3440.938137] Key type .llcrypt unregistered [ 3440.942117] Key type ._llcrypt unregistered [ 3466.293837] Key type ._llcrypt registered [ 3466.298348] Key type .llcrypt registered [ 3466.444840] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3468.053301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3468.222184] alg: No test for adler32 (adler32-zlib) [ 3469.294867] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 3469.672960] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3471.471546] Key type lgssc registered [ 3473.075928] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3482.552133] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3482.608119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3484.312483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3489.882533] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3494.903857] LustreError: 111329:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3494.941594] LustreError: 111329:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3500.001774] LustreError: 111330:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3500.049498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3500.591895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3505.835560] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3515.402116] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3523.376420] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3532.468129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3533.043711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3540.730696] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3551.038955] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3554.569717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3561.586671] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3565.325250] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3569.404471] Lustre: server umount lustre-OST0000 complete [ 3575.268285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3575.277470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3575.299128] Lustre: Skipped 1 previous similar message [ 3580.389207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3580.403659] Lustre: Skipped 2 previous similar messages [ 3583.548979] Lustre: server umount lustre-MDT0000 complete [ 3585.509338] LustreError: 111329:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3585.545893] LustreError: 111329:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3586.273188] Lustre: 110630:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790714108/real 1790714108] req@0000000005764347 x1877699730641792/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790714124 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3586.325838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3587.859565] LustreError: 111309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790714125 with bad export cookie 5971642631454790595 [ 3587.871647] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3587.880192] LustreError: 111309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3588.186313] Lustre: server umount lustre-MDT0001 complete [ 3600.121658] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3603.461995] Key type lgssc unregistered [ 3603.811570] LNet: 114967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3603.827584] LNetError: 114967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3603.842472] LNet: Removed LNI 192.168.204.126@tcp [ 3604.816141] Key type .llcrypt unregistered [ 3604.821132] Key type ._llcrypt unregistered [ 3625.225758] Key type ._llcrypt registered [ 3625.228558] Key type .llcrypt registered [ 3625.407492] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3627.040064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3627.110771] alg: No test for adler32 (adler32-zlib) [ 3628.175228] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 3628.396520] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3630.071161] Key type lgssc registered [ 3631.505933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3639.871317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3639.899207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3641.409937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3645.942809] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3651.555846] LustreError: 116204:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3651.583344] LustreError: 116204:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3655.760972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3656.161514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3661.084997] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3670.339731] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3676.185792] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3683.142670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3683.539191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3686.575539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3689.185342] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3696.748070] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3711.468234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3711.472858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3711.491351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3711.511076] Lustre: Skipped 1 previous similar message [ 3716.579168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3716.596108] Lustre: Skipped 1 previous similar message [ 3716.829515] Lustre: server umount lustre-OST0000 complete [ 3721.696480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3721.710882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3721.715628] Lustre: Skipped 2 previous similar messages [ 3721.732492] Lustre: Skipped 1 previous similar message [ 3726.820689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3726.833972] Lustre: Skipped 1 previous similar message [ 3728.017963] Lustre: server umount lustre-MDT0000 complete [ 3731.937920] LustreError: 116204:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3731.951936] LustreError: 116204:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3732.103492] LustreError: 116184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790714270 with bad export cookie 3141684304420529998 [ 3732.108665] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3732.121114] LustreError: 116184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3732.445655] Lustre: server umount lustre-MDT0001 complete [ 3743.889975] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3747.082971] Key type lgssc unregistered [ 3747.373452] LNet: 119761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3747.386421] LNetError: 119761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3747.399711] LNet: Removed LNI 192.168.204.126@tcp [ 3748.483603] Key type .llcrypt unregistered [ 3748.489174] Key type ._llcrypt unregistered [ 3776.951810] Key type ._llcrypt registered [ 3776.956910] Key type .llcrypt registered [ 3777.146102] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3778.455648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3778.485296] alg: No test for adler32 (adler32-zlib) [ 3780.059151] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 3780.438598] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3782.207455] Key type lgssc registered [ 3783.876707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3794.250457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3794.288655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3795.864573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3801.726488] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3805.155473] LustreError: 121197:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3805.192129] LustreError: 121197:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3810.284556] LustreError: 121196:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3811.535544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3811.968125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3817.221307] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3825.746895] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3832.496445] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3840.450360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3840.960700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3841.993890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3847.440173] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3855.626829] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3862.438462] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3866.725648] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3866.742237] Lustre: Skipped 2 previous similar messages [ 3871.685115] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3871.690833] Lustre: Skipped 2 previous similar messages [ 3872.361397] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3872.427161] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3878.627239] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.26@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3878.634606] Lustre: Skipped 2 previous similar messages [ 3880.014060] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.26@tcp 192.168.0.[2,10]@tcp [ 3886.824482] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.26/24@tcp in log lustre-MDT0000 [ 3886.831133] Lustre: Skipped 2 previous similar messages [ 3886.905394] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.26/24@tcp [ 3886.915723] Lustre: Skipped 1 previous similar message [ 3896.804679] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3896.807805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3896.818096] LustreError: Skipped 1 previous similar message [ 3896.842486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3897.316723] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3897.324045] LustreError: Skipped 1 previous similar message [ 3897.336934] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3897.354085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3897.362401] Lustre: Skipped 1 previous similar message [ 3902.300940] Lustre: server umount lustre-OST0000 complete [ 3907.553847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3907.558037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3907.566922] Lustre: Skipped 1 previous similar message [ 3907.578805] Lustre: Skipped 1 previous similar message [ 3912.562153] Lustre: server umount lustre-MDT0000 complete [ 3912.680140] LustreError: 123779:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3912.690743] LustreError: 123779:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3916.781413] LustreError: 123040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790714454 with bad export cookie 16570361398661911059 [ 3916.796171] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3916.796759] LustreError: 123040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3917.303660] Lustre: server umount lustre-MDT0001 complete [ 3929.944620] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3933.814946] Key type lgssc unregistered [ 3934.353783] LNet: 125791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3934.376264] LNetError: 125791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3934.408160] LNet: Removed LNI 192.168.204.126@tcp [ 3935.183407] Key type .llcrypt unregistered [ 3935.185562] Key type ._llcrypt unregistered [ 3965.893748] Key type ._llcrypt registered [ 3965.899215] Key type .llcrypt registered [ 3966.043767] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3967.378351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3967.419202] alg: No test for adler32 (adler32-zlib) [ 3968.467717] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 3968.694627] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3970.383479] Key type lgssc registered [ 3971.578140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3979.729257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3979.753337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3981.193826] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3981.230676] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.26/24@tcp [ 3981.264071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3985.951464] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3991.525778] LustreError: 127383:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3991.542037] LustreError: 127383:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3994.640401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3994.903456] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3994.934150] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.26/24@tcp [ 3994.987540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3998.896177] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4006.006577] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4012.683551] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4020.973044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4021.260198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4026.364542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4028.684443] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4036.106232] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4051.947165] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4051.951580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4051.966574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4055.008579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4055.015731] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4055.015997] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4055.025085] Lustre: Skipped 1 previous similar message [ 4056.790801] Lustre: server umount lustre-OST0000 complete [ 4065.249968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4065.258522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4065.276618] Lustre: Skipped 1 previous similar message [ 4065.315978] Lustre: Skipped 1 previous similar message [ 4068.336112] Lustre: server umount lustre-MDT0000 complete [ 4070.372122] LustreError: 127377:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4070.417234] LustreError: 127377:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4073.152092] LustreError: 130222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790714611 with bad export cookie 17291506240472520056 [ 4073.160210] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4073.171955] LustreError: 130222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4073.867856] Lustre: server umount lustre-MDT0001 complete [ 4086.390690] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 4089.438335] Key type lgssc unregistered [ 4089.847370] LNet: 130975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4089.876325] LNetError: 130975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4089.900645] LNet: Removed LNI 192.168.204.126@tcp [ 4091.112043] Key type .llcrypt unregistered [ 4091.116422] Key type ._llcrypt unregistered [ 4124.349314] Key type ._llcrypt registered [ 4124.355828] Key type .llcrypt registered [ 4124.472856] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 4141.743386] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4142.958631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4143.047369] alg: No test for adler32 (adler32-zlib) [ 4144.281209] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 4144.508796] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 4146.255205] Key type lgssc registered [ 4147.537699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4185.961936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4185.985261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4187.343597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4187.391548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4187.457150] Lustre: lustre-MDT0000: new disk, initializing [ 4187.553709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4187.586235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4191.812565] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4204.665369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4204.854566] Lustre: 134398:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4204.922510] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4204.928590] Lustre: Skipped 1 previous similar message [ 4204.998231] Lustre: lustre-MDT0001: new disk, initializing [ 4205.121111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4205.178725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4205.200473] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4210.733105] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4220.710369] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4227.504604] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4235.457404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4235.805975] Lustre: lustre-OST0000: new disk, initializing [ 4235.815368] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4235.825750] Lustre: 136364:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4235.972955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4242.422846] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4245.055695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4245.072951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4245.179462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4252.643688] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4258.061497] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4258.305899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4263.543070] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4263.771370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4265.964809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4265.965605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4265.973212] LustreError: Skipped 1 previous similar message [ 4265.984843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4265.994817] Lustre: Skipped 1 previous similar message [ 4266.024616] Lustre: Skipped 1 previous similar message [ 4271.075997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4271.087246] Lustre: Skipped 1 previous similar message [ 4276.193644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4276.202289] Lustre: Skipped 1 previous similar message [ 4280.287203] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4280.582918] Lustre: server umount lustre-OST0000 complete [ 4286.434143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4286.452303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4286.460556] Lustre: Skipped 1 previous similar message [ 4290.627779] Lustre: server umount lustre-MDT0000 complete [ 4294.312083] LustreError: 134389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790714832 with bad export cookie 9473396835085617960 [ 4294.313412] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4294.332368] LustreError: 134389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4294.723373] Lustre: server umount lustre-MDT0001 complete [ 4304.353543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4304.900125] LustreError: 138583:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4304.928350] LustreError: 138583:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4304.990576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4309.930034] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4309.993367] LustreError: 138584:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4314.080567] LustreError: 138583:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4319.202487] LustreError: 138584:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4319.451945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4320.021474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4325.055235] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4335.417649] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4342.829411] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4351.747375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4352.289635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4359.956371] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4368.799464] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4374.177643] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4375.561378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4380.876627] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4381.112517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4389.901385] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4390.073294] Lustre: lustre-OST0001: new disk, initializing [ 4390.077981] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4390.084170] Lustre: 141964:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4390.199029] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4397.155918] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4397.662504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4397.683858] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4397.799372] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4406.507483] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4406.772879] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4412.243103] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4412.559249] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4420.075228] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9cf3090ac800.ost_server_uuid 50 [ 4422.126922] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9cf3090ac800.ost_server_uuid in IDLE state after 0 sec [ 4432.405773] Lustre: server umount lustre-OST0001 complete [ 4433.898154] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4433.908353] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4433.925474] LustreError: 140407:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4433.930017] Lustre: Skipped 2 previous similar messages [ 4433.950692] LustreError: 140407:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4436.450441] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4436.450659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4436.460437] LustreError: Skipped 2 previous similar messages [ 4436.478747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4436.511685] Lustre: Skipped 1 previous similar message [ 4442.484368] Lustre: server umount lustre-OST0000 complete [ 4448.233105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4448.243092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4448.262561] Lustre: Skipped 1 previous similar message [ 4448.272422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4448.278893] Lustre: Skipped 2 previous similar messages [ 4452.771490] Lustre: server umount lustre-MDT0000 complete [ 4454.375403] LustreError: 140327:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4454.402428] LustreError: 140327:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4456.838746] LustreError: 138563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790714994 with bad export cookie 9473396835085618835 [ 4456.844402] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4456.848953] LustreError: 138563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4457.144964] Lustre: server umount lustre-MDT0001 complete [ 4481.185230] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 4484.498943] Key type lgssc unregistered [ 4484.928658] LNet: 144969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4484.942720] LNetError: 144969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4484.969865] LNet: Removed LNI 192.168.204.126@tcp [ 4485.840960] Key type .llcrypt unregistered [ 4485.843424] Key type ._llcrypt unregistered [ 4534.103709] Key type ._llcrypt registered [ 4534.105874] Key type .llcrypt registered [ 4534.214803] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 4551.555078] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4552.971334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4552.997157] alg: No test for adler32 (adler32-zlib) [ 4554.130090] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 4554.573156] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 4556.271304] Key type lgssc registered [ 4557.717803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4601.845797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4601.885134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4603.242563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4603.315265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4603.442752] Lustre: lustre-MDT0000: new disk, initializing [ 4603.516542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4603.542516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4608.967597] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4621.614924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4621.818579] Lustre: 149145:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4621.896626] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4621.901127] Lustre: Skipped 1 previous similar message [ 4621.983358] Lustre: lustre-MDT0001: new disk, initializing [ 4622.104520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4622.167067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4622.187228] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4627.447464] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4639.301410] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4646.290632] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4654.321860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4654.635644] Lustre: lustre-OST0000: new disk, initializing [ 4654.643877] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4654.655928] Lustre: 151114:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4654.756830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4661.848828] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4664.381771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4664.397201] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4664.572274] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4673.688596] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4680.135878] Lustre: Failing over lustre-OST0000 [ 4680.160718] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4680.170665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4680.181786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4680.182422] Lustre: Skipped 1 previous similar message [ 4680.336707] Lustre: server umount lustre-OST0000 complete [ 4701.166755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4701.584943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4701.612551] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4708.688775] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4709.241531] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4713.807667] Lustre: Failing over lustre-MDT0000 [ 4713.868853] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 4713.879830] Lustre: Skipped 1 previous similar message [ 4714.117733] Lustre: server umount lustre-MDT0000 complete [ 4714.465577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4714.476540] LustreError: Skipped 1 previous similar message [ 4714.495482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4716.007544] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4716.022648] Lustre: Skipped 1 previous similar message [ 4721.057259] LustreError: 150513:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4721.088406] LustreError: 150513:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 4724.040336] LustreError: 151122:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4726.246849] LustreError: 149152:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4726.288339] LustreError: 149152:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4729.167465] LustreError: 151122:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4731.360495] Lustre: 147360:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790715253/real 1790715253] req@00000000d7e3b0c2 x1877700868355968/t0(0) o400->MGC192.168.204.126@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790715269 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4731.411183] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4734.288071] LustreError: 150203:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4734.323909] LustreError: 150203:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4737.300037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4741.610580] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbce0e82581b1d558 [ 4741.619852] Lustre: MGC192.168.204.126@tcp: Connection restored to 0@lo (at 0@lo) [ 4742.033484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4742.216368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4742.337588] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4745.548672] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4747.261975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4747.299931] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4747.412127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4748.085586] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4757.478239] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4757.489920] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4757.530260] Lustre: Skipped 1 previous similar message [ 4757.531081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4757.540309] Lustre: Skipped 1 previous similar message [ 4762.639560] Lustre: server umount lustre-OST0000 complete [ 4767.713333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4767.715446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4767.716022] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4767.725438] Lustre: Skipped 1 previous similar message [ 4767.746316] Lustre: Skipped 1 previous similar message [ 4772.833576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4772.838631] Lustre: Skipped 1 previous similar message [ 4772.930486] Lustre: server umount lustre-MDT0000 complete [ 4776.612270] LustreError: 149139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790715314 with bad export cookie 13610133321700988248 [ 4776.626382] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4776.917963] Lustre: server umount lustre-MDT0001 complete [ 4788.594111] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 4791.243443] Key type lgssc unregistered [ 4791.569027] LNet: 154774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4791.579664] LNetError: 154774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4791.606932] LNet: Removed LNI 192.168.204.126@tcp [ 4792.303904] Key type .llcrypt unregistered [ 4792.305285] Key type ._llcrypt unregistered [ 4831.161835] Key type ._llcrypt registered [ 4831.163662] Key type .llcrypt registered [ 4831.274866] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4832.216279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4832.407590] alg: No test for adler32 (adler32-zlib) [ 4833.562535] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 4833.804285] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 4835.551319] Key type lgssc registered [ 4836.911556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4844.835726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4844.855240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4846.490938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4851.452629] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4856.800759] LustreError: 156310:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4856.827786] LustreError: 156310:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4861.920567] LustreError: 156311:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4863.113913] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4867.045797] LustreError: 156310:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4867.560649] LustreError: 156339:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4867.571772] LustreError: 156339:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4867.579211] LustreError: 156339:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 4867.952804] Lustre: server umount lustre-MDT0000 complete [ 4883.094402] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 4895.990556] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4945.580606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4945.844051] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4945.884127] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4945.994967] Lustre: lustre-MDT0000: new disk, initializing [ 4946.114518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4946.149229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4951.489342] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4964.182947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4964.328420] Lustre: 159648:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4964.395467] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4964.399742] Lustre: Skipped 1 previous similar message [ 4964.475467] Lustre: lustre-MDT0001: new disk, initializing [ 4964.607488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4964.645798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4964.660227] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4970.350778] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4981.727397] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4989.162209] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4998.105031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4998.595781] Lustre: lustre-OST0000: new disk, initializing [ 4998.612324] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4998.636046] Lustre: 161617:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4998.856642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5004.901393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5004.912206] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5005.118374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5006.391401] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5018.177166] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5022.789321] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5025.126636] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 16:59:22 (1790715562) [ 5312.992362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5313.007394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5313.027842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5315.552495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5315.552968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5315.571636] Lustre: Skipped 1 previous similar message [ 5315.587604] Lustre: Skipped 2 previous similar messages [ 5318.506487] Lustre: server umount lustre-MDT0000 complete [ 5320.671960] LustreError: 161017:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5320.691228] LustreError: 161017:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5322.700935] LustreError: 161616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790715860 with bad export cookie 14557504502327751444 [ 5322.716606] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5325.791862] LustreError: 163403:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5325.793677] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5325.844836] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5327.844219] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5332.969566] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5337.055127] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5337.195503] Lustre: server umount lustre-MDT0001 complete [ 5349.324742] Lustre: server umount lustre-OST0000 complete [ 5362.601456] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 5375.409140] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5418.575294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5418.897941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5418.971489] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5419.060882] Lustre: lustre-MDT0000: new disk, initializing [ 5419.197080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5419.218623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5424.354630] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5435.432678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5435.572592] Lustre: 166304:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5435.649654] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5435.662638] Lustre: Skipped 1 previous similar message [ 5435.756625] Lustre: lustre-MDT0001: new disk, initializing [ 5435.902498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5435.939910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5435.949992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5440.897150] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5451.153576] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5457.714685] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5466.607149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5466.956964] Lustre: lustre-OST0000: new disk, initializing [ 5466.960405] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5466.965483] Lustre: 168273:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5467.127191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5474.827223] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5475.976916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5476.001275] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5476.108373] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5486.813583] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5491.812967] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5492.091029] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5497.749841] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5498.027492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5501.922929] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5501.924370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5501.931342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5501.957129] Lustre: Skipped 1 previous similar message [ 5506.480519] Lustre: server umount lustre-OST0000 complete [ 5512.162043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5512.200873] Lustre: Skipped 1 previous similar message [ 5516.527966] Lustre: server umount lustre-MDT0000 complete [ 5517.280459] LustreError: 166309:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5517.314977] LustreError: 166309:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5521.543531] LustreError: 166295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790716059 with bad export cookie 14557504502327910274 [ 5521.545545] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5521.564331] LustreError: 166295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5522.262540] Lustre: server umount lustre-MDT0001 complete [ 5534.040739] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:07:51 (1790716071) [ 5554.556752] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 5567.259453] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5609.009969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5609.296871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5609.340884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5609.442748] Lustre: lustre-MDT0000: new disk, initializing [ 5609.530333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5609.553289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5615.426655] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5627.917645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5628.089826] Lustre: 173033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5628.480965] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5633.989223] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5644.455460] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5650.430228] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5658.928033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5659.323901] Lustre: lustre-OST0000: new disk, initializing [ 5659.328322] Lustre: Skipped 1 previous similar message [ 5659.338742] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5659.346228] Lustre: Skipped 2 previous similar messages [ 5659.358075] Lustre: 175011:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5667.369831] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5668.955454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5668.970229] Lustre: Skipped 1 previous similar message [ 5668.974244] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5669.097682] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5677.712457] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5684.711053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5684.712388] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5684.713080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5684.713085] Lustre: Skipped 3 previous similar messages [ 5684.724943] Lustre: Skipped 1 previous similar message [ 5684.753370] LustreError: Skipped 1 previous similar message [ 5690.327200] Lustre: server umount lustre-OST0000 complete [ 5700.896138] Lustre: server umount lustre-MDT0000 complete [ 5704.604207] LustreError: 175915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790716242 with bad export cookie 14557504502327911156 [ 5704.608573] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5704.611921] LustreError: 175915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5704.939529] Lustre: server umount lustre-MDT0001 complete [ 5714.547685] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:10:52 (1790716252) [ 5732.421920] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 5744.381588] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5782.954816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5783.274560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5783.297565] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5783.393436] Lustre: lustre-MDT0000: new disk, initializing [ 5783.493588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5783.505913] Lustre: Skipped 2 previous similar messages [ 5783.521399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5788.427923] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5799.809347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5799.957523] Lustre: 179337:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5800.172569] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5804.804725] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5816.171248] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5823.029623] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5831.295856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5831.657414] Lustre: 181315:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5836.869384] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5837.029378] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5839.139979] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5850.497316] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5862.888349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5862.892679] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5862.908972] Lustre: Skipped 2 previous similar messages [ 5862.910359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5862.948624] Lustre: Skipped 7 previous similar messages [ 5864.169716] Lustre: server umount lustre-OST0000 complete [ 5882.338424] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5882.651281] Lustre: server umount lustre-MDT0000 complete [ 5886.753622] LustreError: 179328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790716424 with bad export cookie 14557504502327912234 [ 5886.758103] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5886.771743] LustreError: 179328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5887.139671] Lustre: server umount lustre-MDT0001 complete [ 5898.310822] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5900.963043] Key type lgssc unregistered [ 5901.312964] LNet: 183170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5901.323384] LNetError: 183170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5901.343587] LNet: Removed LNI 192.168.204.126@tcp [ 5902.275720] Key type .llcrypt unregistered [ 5902.278369] Key type ._llcrypt unregistered [ 5926.709869] Key type ._llcrypt registered [ 5926.712117] Key type .llcrypt registered [ 5926.817751] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5927.711858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5927.850059] alg: No test for adler32 (adler32-zlib) [ 5928.959410] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 5929.241872] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5930.975334] Key type lgssc registered [ 5932.119722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5940.133858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5940.170435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5941.743117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5946.352923] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5951.971365] LustreError: 184601:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5951.993197] LustreError: 184601:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5955.634983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5956.138107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5961.179946] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5969.453325] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5975.845286] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5982.736314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5983.081915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5986.168691] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5988.971222] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5997.053804] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6021.096015] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6021.100179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6021.112511] Lustre: Skipped 1 previous similar message [ 6021.113167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6021.117709] Lustre: Skipped 1 previous similar message [ 6024.512871] Lustre: server umount lustre-OST0000 complete [ 6031.332284] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6031.362701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6031.373474] Lustre: Skipped 1 previous similar message [ 6036.450957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6036.460802] Lustre: Skipped 1 previous similar message [ 6041.570542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6041.574063] Lustre: Skipped 1 previous similar message [ 6043.103160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6043.310052] Lustre: server umount lustre-MDT0000 complete [ 6046.689355] LustreError: 185329:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6046.730387] LustreError: 185329:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6047.372876] LustreError: 184582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790716585 with bad export cookie 12541749756706117847 [ 6047.386172] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6047.795423] Lustre: server umount lustre-MDT0001 complete [ 6060.509471] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 6063.312842] Key type lgssc unregistered [ 6063.638696] LNet: 188161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6063.646042] LNetError: 188161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6063.665211] LNet: Removed LNI 192.168.204.126@tcp [ 6064.506391] Key type .llcrypt unregistered [ 6064.514169] Key type ._llcrypt unregistered [ 6086.466092] Key type ._llcrypt registered [ 6086.468256] Key type .llcrypt registered [ 6086.564445] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6087.325528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6087.394235] alg: No test for adler32 (adler32-zlib) [ 6088.402809] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 6088.667070] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 6090.311418] Key type lgssc registered [ 6091.338778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6099.557841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6099.575834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6101.067437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6105.854292] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6110.180753] LustreError: 189596:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6110.202400] LustreError: 189596:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6115.001473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6115.354561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6120.806539] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6128.968272] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6134.687167] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6140.580154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6140.923276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6146.043461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6147.491103] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6156.579770] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6166.502415] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6166.512071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6166.519202] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6166.544239] Lustre: Skipped 1 previous similar message [ 6168.421630] Lustre: lustre-OST0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 6168.429276] Lustre: Skipped 1 previous similar message [ 6169.911914] Lustre: server umount lustre-OST0000 complete [ 6177.624200] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9cf310bad000.ost_server_uuid 50 [ 6179.311605] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6197.219055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6197.220070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6197.225314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6197.228764] Lustre: Skipped 1 previous similar message [ 6199.787353] Lustre: server umount lustre-MDT0000 complete [ 6202.346995] LustreError: 189596:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6202.358093] LustreError: 189596:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6203.641239] LustreError: 189577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790716741 with bad export cookie 13324749723754167779 [ 6203.649210] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6203.651511] LustreError: 189577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6204.009774] Lustre: server umount lustre-MDT0001 complete [ 6213.237225] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 17:19:10 (1790716750) [ 6220.530087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6221.338839] LustreError: 193510:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6221.478968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6226.923498] LustreError: 193511:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6226.949152] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6232.034261] LustreError: 193510:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6237.304875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6237.793222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6242.910740] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6251.618303] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6258.211435] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6266.713168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6267.214287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6274.646831] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6283.314200] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6290.847826] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6291.042417] Lustre: lustre-OST0001: new disk, initializing [ 6291.046430] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6291.052411] Lustre: 196352:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6291.145575] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6292.156089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6297.032504] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6299.883515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6299.923543] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6300.082234] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6308.464441] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6312.931359] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6312.934982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6312.941849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6312.945040] Lustre: Skipped 1 previous similar message [ 6318.797900] Lustre: server umount lustre-OST0000 complete [ 6319.957656] LustreError: 196413:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6319.976725] LustreError: 196413:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6327.017221] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6327.268748] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6332.569164] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6332.869174] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6335.972097] LustreError: 195335:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6335.996064] LustreError: 195335:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 6351.361745] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6351.362977] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6351.379403] Lustre: Skipped 1 previous similar message [ 6351.391645] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6351.393733] LustreError: Skipped 1 previous similar message [ 6351.407943] Lustre: Skipped 5 previous similar messages [ 6353.379184] Lustre: server umount lustre-OST0001 complete [ 6360.546390] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6360.579656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6360.603336] Lustre: Skipped 1 previous similar message [ 6360.615223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6364.639249] Lustre: server umount lustre-MDT0000 complete [ 6368.917380] LustreError: 193491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790716906 with bad export cookie 13324749723754168850 [ 6368.926913] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6368.931343] LustreError: 193491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6369.358936] Lustre: server umount lustre-MDT0001 complete [ 6396.808992] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 17:22:14 (1790716934) [ 6404.449395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6404.761436] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6404.810974] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6405.010669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6410.480055] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6420.530976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6420.655837] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6420.720461] Lustre: 199797:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6420.808519] LustreError: 199808:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6420.839587] LustreError: 199808:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 6426.833150] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6436.769822] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6443.231401] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6451.046379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6451.284659] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6459.129932] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6459.546849] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6467.514580] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6475.399541] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6475.487095] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6475.925537] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6475.937497] Lustre: Skipped 2 previous similar messages [ 6481.759723] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6491.863794] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6495.715632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6495.720468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6495.727383] Lustre: Skipped 1 previous similar message [ 6495.731745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6495.735415] Lustre: Skipped 2 previous similar messages [ 6501.925757] Lustre: server umount lustre-OST0000 complete [ 6503.249908] LustreError: 202048:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6518.752224] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6518.757972] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6518.765098] Lustre: Skipped 2 previous similar messages [ 6524.392773] Lustre: server umount lustre-OST0001 complete [ 6532.079624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6532.085043] Lustre: Skipped 8 previous similar messages [ 6535.633906] Lustre: server umount lustre-MDT0000 complete [ 6540.490203] LustreError: 199790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790717078 with bad export cookie 13324749723754170257 [ 6540.493315] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6540.496827] LustreError: 199790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6540.895604] Lustre: server umount lustre-MDT0001 complete [ 6567.466876] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 17:25:05 (1790717105) [ 6587.551148] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 6596.963558] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6638.694430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6639.077542] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6639.138562] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6639.246579] Lustre: lustre-MDT0000: new disk, initializing [ 6639.371910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6639.413442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6645.015047] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6656.879438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6657.085988] Lustre: 207891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6657.165908] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6657.171438] Lustre: Skipped 1 previous similar message [ 6657.288856] Lustre: lustre-MDT0001: new disk, initializing [ 6657.472259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6657.494780] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6663.029474] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6673.387888] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6679.780895] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6687.912941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6688.270677] Lustre: lustre-OST0000: new disk, initializing [ 6688.274667] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6688.285830] Lustre: 209863:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6693.495123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6693.512668] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6693.698203] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6696.330500] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6708.429917] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6715.432803] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6715.756359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6722.345591] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6722.651336] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6729.724364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6729.726924] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6729.749173] Lustre: Skipped 2 previous similar messages [ 6729.757275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6729.758296] LustreError: Skipped 2 previous similar messages [ 6729.785774] Lustre: Skipped 5 previous similar messages [ 6731.585375] Lustre: server umount lustre-OST0000 complete [ 6742.995600] Lustre: server umount lustre-MDT0000 complete [ 6745.056892] LustreError: 209029:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6745.106183] LustreError: 209029:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 6747.591638] LustreError: 207884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790717285 with bad export cookie 13324749723754171818 [ 6747.596567] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6747.601413] LustreError: 207884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6748.058702] Lustre: server umount lustre-MDT0001 complete [ 6758.955803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6765.707243] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6776.644078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6783.022795] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6791.951991] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6798.455791] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6806.318351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6813.913336] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6822.669981] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6827.844527] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6830.236781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 6835.631328] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6835.904757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6839.792041] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6839.796081] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6839.811816] LustreError: Skipped 1 previous similar message [ 6839.848823] Lustre: Skipped 4 previous similar messages [ 6844.648695] Lustre: server umount lustre-OST0000 complete [ 6853.367957] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6859.443758] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6889.882768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6896.740449] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6904.627767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6904.639854] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6904.643799] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6904.672730] Lustre: Skipped 1 previous similar message [ 6904.933366] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6907.803721] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6915.062215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6915.078513] Lustre: Skipped 4 previous similar messages [ 6926.303171] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6926.414562] Lustre: server umount lustre-OST0000 complete [ 6937.039352] Lustre: server umount lustre-MDT0000 complete [ 6941.694989] LustreError: 212061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790717479 with bad export cookie 13324749723754172700 [ 6941.702988] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6941.704489] LustreError: 212061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6941.972108] Lustre: server umount lustre-MDT0001 complete [ 6952.848344] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 17:31:30 (1790717490) [ 6962.092610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6962.778326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6962.781550] Lustre: Skipped 6 previous similar messages [ 6968.449801] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6979.434885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6986.507800] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6995.946813] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7002.470569] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7011.060266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7018.458183] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7027.223972] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7033.259339] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7033.502993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7038.304917] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7038.463821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7045.656914] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7045.827839] Lustre: lustre-OST0001: new disk, initializing [ 7045.832862] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7045.839953] Lustre: 221441:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7052.995675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7053.015799] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7053.213536] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7054.024815] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7065.087767] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7069.707792] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7070.030260] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7074.201346] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7074.483700] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7078.884535] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7078.886482] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7078.894461] LustreError: Skipped 4 previous similar messages [ 7078.923048] Lustre: Skipped 5 previous similar messages [ 7082.661791] Lustre: server umount lustre-OST0001 complete [ 7084.005542] LustreError: 219893:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7084.026968] LustreError: 219893:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 7088.885675] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7089.093858] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7093.643060] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7093.798977] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7124.259819] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7130.092607] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7130.105512] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7130.107695] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7130.125968] Lustre: Skipped 1 previous similar message [ 7132.284564] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7142.422648] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7144.437251] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7152.934550] Lustre: server umount lustre-OST0001 complete [ 7176.162875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7176.185042] Lustre: Skipped 15 previous similar messages [ 7185.375709] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7185.692452] Lustre: server umount lustre-OST0000 complete [ 7196.767194] Lustre: server umount lustre-MDT0000 complete [ 7201.294652] LustreError: 218049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790717739 with bad export cookie 13324749723754173666 [ 7201.296333] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7201.305784] LustreError: 218049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7207.870612] Lustre: server umount lustre-MDT0001 complete [ 7236.518624] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 17:36:14 (1790717774) [ 7244.567311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7244.975835] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7245.062833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7251.490098] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7262.427305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7262.641962] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7262.671235] Lustre: 226690:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7269.074907] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7278.577768] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7286.302043] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7294.787306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7295.068198] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7302.331545] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7310.619322] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7322.657518] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7322.823676] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7331.153436] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7340.382856] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7342.271544] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7347.841614] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7348.126280] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7353.617263] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7353.893358] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7359.983381] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9cf30a72c800.ost_server_uuid 50 [ 7361.870624] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9cf30a72c800.ost_server_uuid in IDLE state after 0 sec [ 7363.097987] Lustre: Permanently deactivating lustre-OST0001 [ 7363.151036] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7367.849516] Lustre: Permanently reactivating lustre-OST0001 [ 7367.866496] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7367.871966] Lustre: Skipped 2 previous similar messages [ 7373.819676] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7373.819972] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7373.834638] LustreError: Skipped 4 previous similar messages [ 7373.856640] Lustre: Skipped 7 previous similar messages [ 7378.084968] Lustre: server umount lustre-OST0001 complete [ 7388.764898] Lustre: server umount lustre-OST0000 complete [ 7399.858103] Lustre: server umount lustre-MDT0000 complete [ 7404.776102] LustreError: 229565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790717942 with bad export cookie 13324749723754174954 [ 7404.783844] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7404.796676] LustreError: 229565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7405.245994] Lustre: server umount lustre-MDT0001 complete [ 7432.494846] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 17:39:29 (1790717969) [ 7442.483117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7443.028056] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7443.117374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7443.140741] Lustre: Skipped 2 previous similar messages [ 7449.472206] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7460.848885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7461.009048] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7461.062574] Lustre: 233387:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7467.094909] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7476.926826] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7484.054478] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7492.761957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7493.064682] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7493.095791] Lustre: Permanently deactivating lustre-OST0000 [ 7493.108161] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7493.309955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7493.316366] Lustre: Skipped 10 previous similar messages [ 7500.361462] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7503.399263] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7507.893303] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7514.959195] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7515.060466] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7522.415133] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7525.085735] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7530.629110] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7532.574555] Lustre: Permanently reactivating lustre-OST0000 [ 7532.591657] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7532.602547] Lustre: Skipped 2 previous similar messages [ 7532.987828] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7532.997200] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7533.007492] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7533.072399] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7533.092411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7555.332233] Lustre: server umount lustre-OST0001 complete [ 7565.085629] Lustre: server umount lustre-OST0000 complete [ 7575.803795] Lustre: server umount lustre-MDT0000 complete [ 7580.982561] LustreError: 235257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790718118 with bad export cookie 13324749723754176592 [ 7580.985842] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7580.993062] LustreError: 235257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7581.267192] Lustre: server umount lustre-MDT0001 complete [ 7594.765081] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7598.128370] Key type lgssc unregistered [ 7598.394754] LNet: 238272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7598.400593] LNetError: 238272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7598.423753] LNet: Removed LNI 192.168.204.126@tcp [ 7599.265919] Key type .llcrypt unregistered [ 7599.272394] Key type ._llcrypt unregistered [ 7624.073113] Key type ._llcrypt registered [ 7624.074932] Key type .llcrypt registered [ 7624.164995] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7625.210730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7625.250691] alg: No test for adler32 (adler32-zlib) [ 7626.330944] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 7626.583706] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 7628.311222] Key type lgssc registered [ 7629.674792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7640.489274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7640.519732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7641.962946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7645.423713] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7651.317327] LustreError: 239757:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7651.355785] LustreError: 239757:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7656.142636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7656.328200] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7656.387185] Lustre: 239743:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7656.402272] Lustre: Permanently deactivating lustre-MDT0001 [ 7656.429088] LustreError: 239756:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7656.449584] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7656.930646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7658.552446] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7658.572363] LustreError: 239787:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7658.601068] LustreError: 239787:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7658.621851] LustreError: 239787:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 1, failed: rc = -108 [ 7663.618857] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7671.992215] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7679.117144] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7687.275413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7687.779317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7695.424692] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7701.095114] LustreError: 241601:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7701.125118] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7705.465335] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7706.606256] LustreError: 241601:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7711.730177] LustreError: 242161:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7711.743692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7711.761402] LustreError: 242161:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7714.242686] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7714.578459] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7719.934816] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7719.940520] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7722.079455] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7731.006378] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7734.079664] Lustre: Permanently reactivating lustre-MDT0001 [ 7734.108276] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7734.119582] Lustre: Skipped 2 previous similar messages [ 7738.178755] Lustre: lustre-MDT0001: Client f5681719-47e4-46e0-8694-1cbf21e012c7 (at 192.168.204.26@tcp) reconnecting [ 7744.671230] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7746.242247] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7752.211238] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7752.451481] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7757.904279] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7758.137115] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7761.155157] Lustre: Permanently deactivating lustre-MDT0001 [ 7761.174631] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7761.182949] Lustre: Skipped 2 previous similar messages [ 7771.244145] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7779.822773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7779.832656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7779.851165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7781.351791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7781.352861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7781.372742] Lustre: Skipped 2 previous similar messages [ 7781.385585] Lustre: Skipped 2 previous similar messages [ 7783.525762] Lustre: server umount lustre-MDT0000 complete [ 7786.468445] LustreError: 239756:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7786.503536] LustreError: 239756:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7787.604599] LustreError: 243306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790718325 with bad export cookie 2849135427824878993 [ 7787.612201] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7787.623634] LustreError: 243306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7787.963440] Lustre: server umount lustre-MDT0001 complete [ 7793.920542] Lustre: server umount lustre-OST0000 complete [ 7804.944971] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 17:45:42 (1790718342) [ 7815.277707] Lustre: server umount lustre-OST0001 complete [ 7824.057135] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 7835.848675] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7873.792706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7874.207976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7874.217197] Lustre: Skipped 2 previous similar messages [ 7874.254918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7874.351966] Lustre: lustre-MDT0000: new disk, initializing [ 7874.446714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7874.464080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7879.202503] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7890.777825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7890.973398] Lustre: 248037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7891.056653] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7891.072458] Lustre: Skipped 1 previous similar message [ 7891.232222] Lustre: lustre-MDT0001: new disk, initializing [ 7891.339443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7891.381165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7891.396452] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7896.920370] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7907.584899] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7915.401882] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7922.733104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7923.116764] Lustre: lustre-OST0000: new disk, initializing [ 7923.128107] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7923.142873] Lustre: 250007:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7923.253237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7928.666709] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7932.501554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7932.512424] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7932.639394] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7939.566707] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7941.862418] LustreError: 248046:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7950.394764] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7950.580615] Lustre: lustre-OST0001: new disk, initializing [ 7950.587684] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7950.603258] Lustre: 251124:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7951.925393] LustreError: 248046:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7951.948412] LustreError: 248045:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7957.415311] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7958.674890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7958.684421] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7958.943188] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7961.968088] LustreError: 248045:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7968.262961] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7970.153513] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7974.369116] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7974.370234] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7974.376846] LustreError: Skipped 1 previous similar message [ 7974.387485] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7974.399123] Lustre: Skipped 1 previous similar message [ 7974.423415] Lustre: Skipped 2 previous similar messages [ 7978.659813] Lustre: server umount lustre-OST0001 complete [ 7979.503886] LustreError: 249990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7979.524935] LustreError: 249990:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7999.458692] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7999.467117] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7999.476267] LustreError: Skipped 1 previous similar message [ 7999.490474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8001.820393] Lustre: server umount lustre-OST0000 complete [ 8009.184444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8009.190428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8009.202047] Lustre: Skipped 1 previous similar message [ 8009.209195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8009.215314] Lustre: Skipped 1 previous similar message [ 8012.275750] Lustre: server umount lustre-MDT0000 complete [ 8015.328330] LustreError: 251546:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8015.359723] LustreError: 251546:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 8016.348716] LustreError: 248029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790718554 with bad export cookie 2849135427824881807 [ 8016.357509] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8022.699295] Lustre: server umount lustre-MDT0001 complete [ 8036.931874] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8039.347652] Key type lgssc unregistered [ 8039.599580] LNet: 253163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8039.605504] LNetError: 253163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8039.616969] LNet: Removed LNI 192.168.204.126@tcp [ 8040.428973] Key type .llcrypt unregistered [ 8040.430535] Key type ._llcrypt unregistered [ 8088.314634] Key type ._llcrypt registered [ 8088.317486] Key type .llcrypt registered [ 8088.479917] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8089.678089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8089.707204] alg: No test for adler32 (adler32-zlib) [ 8090.967411] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 8091.379683] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 8093.090562] Key type lgssc registered [ 8094.460982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8104.067624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8104.105415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8105.492422] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8105.572843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8105.830936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8110.776683] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8120.890234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8121.068926] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8121.121180] Lustre: 255384:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8121.506686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8126.719137] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8136.029043] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8144.293749] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8152.646905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8152.908712] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8153.256507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8158.886731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8161.557706] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8170.401503] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8225.763342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8225.771984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8230.369794] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8230.379786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8230.393938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8231.995468] Lustre: server umount lustre-OST0000 complete [ 8236.889357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8251.527077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8252.015269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8255.077759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8255.091802] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8258.491808] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8266.926599] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8270.443280] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8270.466326] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8311.780831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8311.781846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8311.785441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8311.785450] Lustre: Skipped 1 previous similar message [ 8311.794755] LustreError: Skipped 1 previous similar message [ 8311.823613] Lustre: Skipped 1 previous similar message [ 8313.867402] Lustre: server umount lustre-OST0000 complete [ 8320.999521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8321.020705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8321.050461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8321.061697] Lustre: Skipped 1 previous similar message [ 8324.986679] Lustre: server umount lustre-MDT0000 complete [ 8329.227917] LustreError: 255377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790718867 with bad export cookie 2302623360216150933 [ 8329.242621] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8329.599906] Lustre: server umount lustre-MDT0001 complete [ 8341.097826] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8344.623451] Key type lgssc unregistered [ 8345.018847] LNet: 260817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8345.025639] LNetError: 260817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8345.046798] LNet: Removed LNI 192.168.204.126@tcp [ 8346.010684] Key type .llcrypt unregistered [ 8346.013062] Key type ._llcrypt unregistered [ 8374.618887] Key type ._llcrypt registered [ 8374.620808] Key type .llcrypt registered [ 8374.811124] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8376.024447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8376.176466] alg: No test for adler32 (adler32-zlib) [ 8377.383806] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 8377.787782] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 8379.551222] Key type lgssc registered [ 8380.948615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8388.986678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8389.003817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8390.499706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8395.183679] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8400.867974] LustreError: 262250:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8400.891372] LustreError: 262250:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8405.987436] LustreError: 262249:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8406.701295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8407.314991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8413.167601] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8420.557379] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8428.836139] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8437.823252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8438.133020] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8446.007360] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8454.529842] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8460.673498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8491.514176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8491.516857] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8491.541026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8491.569621] Lustre: Skipped 1 previous similar message [ 8496.612833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8496.629440] Lustre: Skipped 1 previous similar message [ 8496.879588] Lustre: server umount lustre-OST0000 complete [ 8501.730916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8501.734226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8501.752541] Lustre: Skipped 2 previous similar messages [ 8506.856110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8506.861729] Lustre: Skipped 2 previous similar messages [ 8507.791629] Lustre: server umount lustre-MDT0000 complete [ 8511.972590] LustreError: 264694:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8512.001119] LustreError: 264694:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8512.449409] LustreError: 262230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790719050 with bad export cookie 16686328518006929195 [ 8512.456767] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8512.463696] LustreError: 262230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8513.084174] Lustre: server umount lustre-MDT0001 complete [ 8526.006697] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8528.877434] Key type lgssc unregistered [ 8529.275696] LNet: 266578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8529.282412] LNetError: 266578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8529.296884] LNet: Removed LNI 192.168.204.126@tcp [ 8530.221730] Key type .llcrypt unregistered [ 8530.227046] Key type ._llcrypt unregistered [ 8551.327433] Key type ._llcrypt registered [ 8551.329550] Key type .llcrypt registered [ 8551.442842] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8552.810567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8552.904746] alg: No test for adler32 (adler32-zlib) [ 8554.049713] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 8554.312267] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 8556.047684] Key type lgssc registered [ 8557.530268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8566.688374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8566.740757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8568.213904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8573.509326] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8578.536176] LustreError: 267815:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8578.567134] LustreError: 267815:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8583.136541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8583.447849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8588.738955] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8598.110951] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8605.593898] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8614.360357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8614.894491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8621.478484] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8630.821989] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8638.452322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8653.802501] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8653.804267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8653.806355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8653.812909] LustreError: Skipped 1 previous similar message [ 8653.830889] Lustre: Skipped 1 previous similar message [ 8655.195784] Lustre: server umount lustre-OST0000 complete [ 8660.450084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8660.461056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8660.480734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8660.485586] Lustre: Skipped 1 previous similar message [ 8664.036586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8664.039524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8665.706556] Lustre: server umount lustre-MDT0000 complete [ 8669.153977] LustreError: 269590:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8669.181315] LustreError: 269590:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8670.716034] LustreError: 267795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790719208 with bad export cookie 15308590686220647424 [ 8670.729766] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8671.330106] Lustre: server umount lustre-MDT0001 complete [ 8684.352728] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8688.065556] Key type lgssc unregistered [ 8688.443547] LNet: 271515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8688.462477] LNetError: 271515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8689.514877] LNet: Removed LNI 192.168.204.126@tcp [ 8690.490233] Key type .llcrypt unregistered [ 8690.493784] Key type ._llcrypt unregistered [ 8716.679969] Key type ._llcrypt registered [ 8716.684329] Key type .llcrypt registered [ 8716.827521] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8718.087288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8718.249079] alg: No test for adler32 (adler32-zlib) [ 8719.315813] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 8719.584927] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 8721.303383] Key type lgssc registered [ 8723.258899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8733.415692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8733.448428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8735.045672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8739.653821] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8744.428897] LustreError: 272958:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8744.467687] LustreError: 272958:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8748.045155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8748.305186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8752.335350] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8762.026157] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8770.114486] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8778.330670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8778.822758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8779.897564] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8785.451895] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8793.608438] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8820.196858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8820.199725] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8820.210198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8820.220117] Lustre: Skipped 1 previous similar message [ 8820.233439] Lustre: Skipped 1 previous similar message [ 8825.330836] LustreError: 275449:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8825.346454] LustreError: 275449:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8825.449854] Lustre: server umount lustre-OST0000 complete [ 8830.433532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8830.438945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8830.445791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8830.475207] Lustre: Skipped 1 previous similar message [ 8835.475971] Lustre: server umount lustre-MDT0000 complete [ 8835.554256] LustreError: 272955:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8835.579889] LustreError: 272955:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8839.252107] LustreError: 272938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790719377 with bad export cookie 6701717959028277793 [ 8839.264786] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8839.265467] LustreError: 272938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8839.636708] Lustre: server umount lustre-MDT0001 complete [ 8851.547342] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8854.475534] Key type lgssc unregistered [ 8854.820752] LNet: 277265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8854.830908] LNetError: 277265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8854.857088] LNet: Removed LNI 192.168.204.126@tcp [ 8855.869860] Key type .llcrypt unregistered [ 8855.872830] Key type ._llcrypt unregistered [ 8878.803767] Key type ._llcrypt registered [ 8878.806338] Key type .llcrypt registered [ 8878.903033] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8880.313454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8880.342380] alg: No test for adler32 (adler32-zlib) [ 8881.669649] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 8882.097578] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 8883.887215] Key type lgssc registered [ 8885.799972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8893.865290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8893.899748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8895.397296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8900.220483] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8905.697451] LustreError: 278517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8905.713956] LustreError: 278517:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8909.545854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8910.052555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8915.539375] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8924.668994] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8931.168275] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8939.418098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8940.021551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8941.053386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8947.367048] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8956.063920] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8969.701757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8969.704815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8969.724215] Lustre: Skipped 1 previous similar message [ 8973.213677] Lustre: server umount lustre-OST0000 complete [ 8979.941327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8979.955606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8979.961270] Lustre: Skipped 1 previous similar message [ 8979.978115] Lustre: Skipped 2 previous similar messages [ 8984.018819] Lustre: server umount lustre-MDT0000 complete [ 8985.060956] LustreError: 278501:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8985.108763] LustreError: 278501:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8988.191453] LustreError: 278483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790719526 with bad export cookie 1745030011873700390 [ 8988.197496] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8988.215663] LustreError: 278483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8988.589623] Lustre: server umount lustre-MDT0001 complete [ 9001.260814] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 9004.163314] Key type lgssc unregistered [ 9004.528910] LNet: 282219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9004.542358] LNetError: 282219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9004.565925] LNet: Removed LNI 192.168.204.126@tcp [ 9005.595342] Key type .llcrypt unregistered [ 9005.597948] Key type ._llcrypt unregistered [ 9019.313562] Key type ._llcrypt registered [ 9019.315845] Key type .llcrypt registered [ 9019.399347] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9040.114835] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 9057.542519] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9058.145761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9058.153839] alg: No test for adler32 (adler32-zlib) [ 9059.349892] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 9059.615497] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 9061.273190] Key type lgssc registered [ 9062.629405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9101.041130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9101.093364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9102.454370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9102.506181] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9102.636407] Lustre: lustre-MDT0000: new disk, initializing [ 9102.805289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9102.838735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9109.096387] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9121.281432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9121.499516] Lustre: 285789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9121.565185] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9121.568928] Lustre: Skipped 1 previous similar message [ 9121.639308] Lustre: lustre-MDT0001: new disk, initializing [ 9121.730409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9121.764579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9121.777367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9126.776726] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9137.023542] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9143.439959] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9151.018113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9151.310351] Lustre: lustre-OST0000: new disk, initializing [ 9151.316494] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9151.336393] Lustre: 287759:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9151.418922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9158.208514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9158.235626] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9158.334539] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9158.358608] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9168.809808] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9174.255489] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9174.488673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9180.021411] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9180.255303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9184.225146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9184.226847] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9184.232796] LustreError: Skipped 1 previous similar message [ 9184.260735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9187.296543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9187.312520] Lustre: Skipped 2 previous similar messages [ 9188.736953] Lustre: server umount lustre-OST0000 complete [ 9193.446489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9193.460994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9193.487030] Lustre: Skipped 1 previous similar message [ 9193.506683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9197.536180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9197.540178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9198.950568] Lustre: server umount lustre-MDT0000 complete [ 9202.626139] LustreError: 285781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790719740 with bad export cookie 16223459583656633605 [ 9202.631807] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9202.632866] LustreError: 285781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9203.013920] Lustre: server umount lustre-MDT0001 complete [ 9212.490689] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 18:09:09 (1790719749) [ 9219.724996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9220.299158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9225.003342] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9225.699664] LustreError: 290173:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9225.728375] LustreError: 290173:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9230.826672] LustreError: 290172:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9235.939445] LustreError: 290173:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9237.204623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9237.683977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9243.732757] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9251.709325] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9257.667890] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9265.060327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9265.425420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9271.905836] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9279.525329] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9297.376438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9297.394315] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9297.430313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9297.443147] Lustre: Skipped 1 previous similar message [ 9299.886891] Lustre: server umount lustre-OST0000 complete [ 9308.641639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9308.649853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9308.666581] Lustre: Skipped 2 previous similar messages [ 9308.682851] Lustre: Skipped 3 previous similar messages [ 9319.391148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9319.762538] Lustre: server umount lustre-MDT0000 complete [ 9324.005746] LustreError: 290169:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9324.032210] LustreError: 290169:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9324.368928] LustreError: 290154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790719862 with bad export cookie 16223459583656634487 [ 9324.380697] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9324.381935] LustreError: 290154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9324.798456] Lustre: server umount lustre-MDT0001 complete [ 9337.983647] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 9342.022368] Key type lgssc unregistered [ 9342.433704] LNet: 293728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9342.443546] LNetError: 293728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9342.467923] LNet: Removed LNI 192.168.204.126@tcp [ 9343.603883] Key type .llcrypt unregistered [ 9343.609214] Key type ._llcrypt unregistered [ 9378.452940] Key type ._llcrypt registered [ 9378.455675] Key type .llcrypt registered [ 9378.561586] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 9395.054553] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9396.139560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9396.349387] alg: No test for adler32 (adler32-zlib) [ 9397.458056] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 9397.716686] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 9399.392223] Key type lgssc registered [ 9400.653157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9448.243829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9448.278266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9449.470799] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9449.499614] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9449.583339] Lustre: lustre-MDT0000: new disk, initializing [ 9449.651706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9449.672507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9453.488458] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9464.138317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9464.349996] Lustre: 297571:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9464.434189] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9464.440053] Lustre: Skipped 1 previous similar message [ 9464.533806] Lustre: lustre-MDT0001: new disk, initializing [ 9464.664025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9464.708287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9464.717618] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9470.724291] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9482.780779] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9488.735965] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9496.006153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9496.290572] Lustre: lustre-OST2710: new disk, initializing [ 9496.294863] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9496.303849] Lustre: 299541:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9496.374713] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9502.055748] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9504.827939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9504.849231] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9504.980160] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9511.390443] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9519.030591] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9519.204382] Lustre: lustre-OST03e8: new disk, initializing [ 9519.209896] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9519.218473] Lustre: 300605:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9519.341207] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9524.863229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9524.884715] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9525.025481] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9526.335525] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9535.173337] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9542.893941] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9543.158299] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9548.183956] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9548.396492] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9553.472626] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9553.820423] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9558.698578] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9559.034717] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9565.549670] Lustre: server umount lustre-MDT0000 complete [ 9566.185859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9566.206910] Lustre: Skipped 1 previous similar message [ 9569.155525] LustreError: 299540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790720107 with bad export cookie 17465778654833621449 [ 9569.165277] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9569.166393] LustreError: 299540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9569.520060] Lustre: server umount lustre-MDT0001 complete [ 9583.324683] Lustre: server umount lustre-OST2710 complete [ 9597.901715] Lustre: server umount lustre-OST03e8 complete [ 9619.928301] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 9631.169028] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9669.952872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9670.442484] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9670.484840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9670.550658] Lustre: lustre-MDT0000: new disk, initializing [ 9670.651918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9670.680627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9675.385838] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9686.295520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9686.428457] Lustre: 305857:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9686.474501] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9686.479492] Lustre: Skipped 1 previous similar message [ 9686.579480] Lustre: lustre-MDT0001: new disk, initializing [ 9686.673059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9686.718908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9686.728273] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9690.774767] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9700.039987] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9706.106509] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9713.838699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9714.094286] Lustre: lustre-OST0000: new disk, initializing [ 9714.097889] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9714.104404] Lustre: 307823:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9714.200810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9721.871231] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9723.967742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9723.988730] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9724.104215] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9730.848552] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9735.969395] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9736.264251] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9741.355590] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9741.576418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9744.872122] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9744.877056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9744.886049] Lustre: Skipped 2 previous similar messages [ 9744.891115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9744.902324] Lustre: Skipped 1 previous similar message [ 9749.736315] Lustre: server umount lustre-OST0000 complete [ 9755.107870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9755.109638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9755.120558] Lustre: Skipped 2 previous similar messages [ 9755.136808] Lustre: Skipped 1 previous similar message [ 9759.926638] Lustre: server umount lustre-MDT0000 complete [ 9760.226039] LustreError: 305868:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9760.236255] LustreError: 305868:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 9763.776399] LustreError: 305850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790720301 with bad export cookie 17465778654833623136 [ 9763.779556] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9763.790033] LustreError: 305850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9764.057035] Lustre: server umount lustre-MDT0001 complete [ 9772.673800] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 18:18:30 (1790720310) [ 9774.215694] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9776.197858] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 18:18:33 (1790720313) [ 9779.803287] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9806.616674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9806.895671] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9806.921129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9807.153863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9812.146184] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9822.573889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9822.740827] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9822.767805] Lustre: 311677:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9828.246822] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9851.499393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9851.704754] Lustre: Denying initial registration attempt from nid 192.168.204.126@tcp, specified as failover [ 9851.714949] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9851.722489] LustreError: 313133:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9851.731623] LustreError: 313133:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9851.738778] LustreError: 313133:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9851.907419] Lustre: server umount lustre-OST0000 complete [ 9851.914919] LustreError: 313133:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9859.041037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9859.047375] LustreError: Skipped 1 previous similar message [ 9859.051618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9859.061605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9862.628994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9862.639308] Lustre: Skipped 1 previous similar message [ 9864.329423] Lustre: server umount lustre-MDT0000 complete [ 9867.376519] LustreError: 311670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790720405 with bad export cookie 17465778654833624018 [ 9867.380627] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9867.387523] LustreError: 311670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9867.671899] Lustre: server umount lustre-MDT0001 complete [ 9877.781330] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 9880.456921] Key type lgssc unregistered [ 9880.728943] LNet: 314252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9880.738931] LNetError: 314252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9880.751221] LNet: Removed LNI 192.168.204.126@tcp [ 9881.492132] Key type .llcrypt unregistered [ 9881.495345] Key type ._llcrypt unregistered [ 9893.624907] Key type ._llcrypt registered [ 9893.627480] Key type .llcrypt registered [ 9893.746475] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9894.917405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9894.965038] alg: No test for adler32 (adler32-zlib) [ 9895.973345] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 9896.320377] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 9898.071313] Key type lgssc registered [ 9899.597282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9944.090790] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9953.233707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9953.296695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9954.623802] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9954.658491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9954.838454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9959.664612] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9968.138676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9968.251765] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9968.293500] Lustre: 317033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9968.583702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9972.515530] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9979.552988] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9985.413923] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10005.554529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10005.785587] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10006.065375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10011.895939] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10019.006880] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10025.448110] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10025.460213] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10025.472498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10025.970726] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10031.074140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10031.084616] Lustre: Skipped 1 previous similar message [10035.170147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10035.333799] Lustre: server umount lustre-OST0000 complete [10040.289524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10040.296895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10040.310536] Lustre: Skipped 1 previous similar message [10040.319401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10040.322792] Lustre: Skipped 1 previous similar message [10041.313957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10044.720763] Lustre: server umount lustre-MDT0000 complete [10048.802368] LustreError: 317027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790720586 with bad export cookie 12933398303198573717 [10048.813662] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10048.829640] LustreError: 317027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10049.207290] Lustre: server umount lustre-MDT0001 complete [10059.405866] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [10062.117486] Key type lgssc unregistered [10062.495550] LNet: 320606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10062.515589] LNetError: 320606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10062.551989] LNet: Removed LNI 192.168.204.126@tcp [10063.324307] Key type .llcrypt unregistered [10063.329223] Key type ._llcrypt unregistered [10092.162381] Key type ._llcrypt registered [10092.167312] Key type .llcrypt registered [10092.367573] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [10093.321201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10093.371820] alg: No test for adler32 (adler32-zlib) [10094.362899] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [10094.653967] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [10096.383282] Key type lgssc registered [10097.422763] Lustre: Echo OBD driver; http://www.lustre.org/ [10104.949719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10104.969637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10106.383275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10110.528671] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10116.590288] LustreError: 322037:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10116.628781] LustreError: 322037:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10120.670378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10121.175026] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10126.292806] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10135.860217] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10142.160535] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10151.174682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10151.567700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10157.017386] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10163.424896] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10177.506027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10177.524522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10177.536555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10177.541271] Lustre: Skipped 2 previous similar messages [10177.552469] Lustre: Skipped 2 previous similar messages [10180.478285] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [10182.629526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10185.562067] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [10185.571190] Lustre: Skipped 2 previous similar messages [10190.665043] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [10190.687099] Lustre: Skipped 3 previous similar messages [10191.327512] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10191.574501] Lustre: server umount lustre-MDT0000 complete [10192.867370] LustreError: 322033:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10192.909332] LustreError: 322033:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10195.790396] LustreError: 322034:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10196.579909] LustreError: 322018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790720734 with bad export cookie 5500884793984394953 [10196.587108] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10196.602093] LustreError: 322018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10198.005932] LustreError: 322032:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10198.015852] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10200.050971] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10200.068332] Lustre: Skipped 1 previous similar message [10203.395695] Lustre: server umount lustre-MDT0001 complete [10212.583678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10223.267187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10242.018511] LustreError: 321338:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a773cacc x1877706677461760/t0(0) o250->MGC192.168.204.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10242.376996] LustreError: 323872:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10242.388547] LustreError: 323872:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10242.436710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10242.498365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10247.424094] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10252.782218] LustreError: 323873:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10252.811080] LustreError: 323873:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10256.910665] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10257.150709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10257.655622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10262.571947] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10263.040437] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10270.438697] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10271.922571] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10277.949743] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10279.745851] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10288.619649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10288.622733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10288.635861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10288.635870] Lustre: Skipped 2 previous similar messages [10288.668405] LustreError: Skipped 1 previous similar message [10291.510409] Lustre: server umount lustre-OST0000 complete [10298.849811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10298.850691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10298.881652] Lustre: Skipped 2 previous similar messages [10310.111218] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10310.277737] Lustre: server umount lustre-MDT0000 complete [10313.185400] LustreError: 325658:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10313.211125] LustreError: 325658:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10313.486452] LustreError: 322019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790720851 with bad export cookie 5500884793984402548 [10313.488327] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10313.508836] LustreError: 322019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10313.935895] Lustre: server umount lustre-MDT0001 complete [10324.978254] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [10327.543056] Key type lgssc unregistered [10327.812878] LNet: 328331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10327.820992] LNetError: 328331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10327.842590] LNet: Removed LNI 192.168.204.126@tcp [10328.986444] Key type .llcrypt unregistered [10328.989730] Key type ._llcrypt unregistered [10354.394432] Key type ._llcrypt registered [10354.398146] Key type .llcrypt registered [10354.514456] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [10355.694488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10355.799115] alg: No test for adler32 (adler32-zlib) [10356.898184] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [10357.215560] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [10358.927336] Key type lgssc registered [10360.444755] Lustre: Echo OBD driver; http://www.lustre.org/ [10368.093243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10368.123927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10369.573593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10374.031180] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10379.745089] LustreError: 329763:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10379.764734] LustreError: 329763:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10383.282682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10383.759710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10388.118724] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10396.709626] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10401.914595] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10408.546048] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10408.547581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10408.573855] Lustre: Skipped 1 previous similar message [10408.578307] Lustre: Skipped 1 previous similar message [10410.796237] Lustre: server umount lustre-MDT0000 complete [10414.048787] LustreError: 329763:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10414.065193] LustreError: 329763:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10414.125031] LustreError: 329744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790720952 with bad export cookie 2870505302250391904 [10414.127628] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10414.138472] LustreError: 329744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10414.367214] Lustre: server umount lustre-MDT0001 complete [10433.616453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10434.082274] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10434.160028] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10434.405354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10439.191402] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10449.513546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10449.608913] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10449.646287] Lustre: 332353:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10450.008734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10454.720520] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10464.817341] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10472.958904] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10484.533613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10484.885935] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10484.899698] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10484.916696] LustreError: 334236:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [10484.925824] LustreError: 334236:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10484.942154] LustreError: 334236:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10485.058237] Lustre: server umount lustre-OST0000 complete [10485.062293] LustreError: 334236:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10497.720484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10498.027435] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10498.455615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10506.027581] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10507.505848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10514.672825] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10522.517686] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10522.587831] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10522.646842] Lustre: lustre-OST0001: new disk, initializing [10522.649844] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10522.654286] Lustre: 335780:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10522.760871] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10530.099874] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10530.393086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10530.403879] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10530.531879] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10538.232129] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10546.166816] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10546.175853] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10546.181767] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10546.196687] Lustre: Skipped 1 previous similar message [10547.802487] Lustre: server umount lustre-OST0001 complete [10551.290731] LustreError: 336160:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10551.326078] LustreError: 336160:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10553.313630] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10553.323517] LustreError: Skipped 1 previous similar message [10553.330737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10553.347414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10553.354945] Lustre: Skipped 1 previous similar message [10556.390634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10556.393941] LustreError: 334703:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10556.402050] Lustre: Skipped 2 previous similar messages [10556.435651] LustreError: 334703:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10561.510390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10561.518822] LustreError: 334701:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10561.533793] LustreError: 334701:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10566.111192] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10566.307146] Lustre: server umount lustre-OST0000 complete [10571.749668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10571.760447] Lustre: Skipped 1 previous similar message [10571.774859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10571.791561] Lustre: Skipped 2 previous similar messages [10585.055878] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10585.711338] Lustre: server umount lustre-MDT0000 complete [10587.106287] LustreError: 332363:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10587.126144] LustreError: 332363:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10590.333274] LustreError: 332344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790721128 with bad export cookie 2870505302250392457 [10590.335254] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10590.344690] LustreError: 332344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10590.722970] Lustre: server umount lustre-MDT0001 complete [10604.136425] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [10607.653454] Key type lgssc unregistered [10607.976218] LNet: 337737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10607.984429] LNetError: 337737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10608.007724] LNet: Removed LNI 192.168.204.126@tcp [10608.789614] Key type .llcrypt unregistered [10608.791399] Key type ._llcrypt unregistered [10691.456979] Key type ._llcrypt registered [10691.459875] Key type .llcrypt registered [10691.637962] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [10707.326817] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [10708.607144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10708.730484] alg: No test for adler32 (adler32-zlib) [10709.883361] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [10710.208913] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [10711.895539] Key type lgssc registered [10713.582922] Lustre: Echo OBD driver; http://www.lustre.org/ [10754.491178] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10754.546195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10755.831713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10755.875253] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10755.954603] Lustre: lustre-MDT0000: new disk, initializing [10756.022459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10756.048309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10761.704440] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10773.375434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10773.517895] Lustre: 342692:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10773.578228] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10773.582209] Lustre: Skipped 1 previous similar message [10773.659399] Lustre: lustre-MDT0001: new disk, initializing [10773.794053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10773.854060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10779.087354] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10790.797787] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10797.963656] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10805.839725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10806.256821] Lustre: lustre-OST0000: new disk, initializing [10806.272273] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10806.284073] Lustre: 344662:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10806.382245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10813.026214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10813.041814] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10813.197575] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10814.434437] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10824.715668] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10829.855457] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10830.098424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10834.114870] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10834.332080] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10839.008347] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10839.021933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10839.022255] LustreError: Skipped 1 previous similar message [10839.024262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10839.057014] Lustre: Skipped 1 previous similar message [10842.457387] Lustre: server umount lustre-OST0000 complete [10849.259915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10849.273765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10849.278378] Lustre: Skipped 1 previous similar message [10849.297855] Lustre: Skipped 2 previous similar messages [10854.371151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10854.380680] Lustre: Skipped 1 previous similar message [10859.490389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10859.498592] Lustre: Skipped 1 previous similar message [10861.023536] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10861.403263] Lustre: server umount lustre-MDT0000 complete [10865.802062] LustreError: 342685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790721403 with bad export cookie 17174842354896975857 [10865.807516] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10865.816328] LustreError: 342685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10866.160971] Lustre: server umount lustre-MDT0001 complete [10876.185348] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 18:36:53 (1790721413) [10885.980811] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 18:37:03 (1790721423) [10895.434270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10896.035363] LustreError: 347418:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10896.061433] LustreError: 347418:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10896.140945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10901.335653] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10901.475996] LustreError: 347419:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10906.595143] LustreError: 347418:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10910.864260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10911.282684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10915.922252] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10923.966128] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10929.753031] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10936.782987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10937.365434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10944.263240] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10951.112353] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10961.009721] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10963.501485] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10965.492928] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10967.504223] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10969.508952] Lustre: Failing over lustre-MDT0000 [10969.821203] Lustre: server umount lustre-MDT0000 complete [10972.133560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10972.136327] LustreError: 348422:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10972.144827] Lustre: Skipped 1 previous similar message [10977.248858] LustreError: 348778:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10977.263518] LustreError: 348778:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10987.493706] LustreError: 348422:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10987.517460] LustreError: 348422:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [10988.512804] Lustre: 340905:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790721510/real 1790721510] req@000000000facf661 x1877707323416320/t0(0) o400->MGC192.168.204.126@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790721526 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10988.544167] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10989.583693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10998.762191] LustreError: 340902:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000705a1be5 x1877707323422336/t0(0) o250->MGC192.168.204.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10999.164287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10999.231792] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10999.627358] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11004.411337] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11004.456792] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [11004.536259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11004.700236] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11014.972376] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11016.829705] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11020.416656] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11022.622198] Lustre: Failing over lustre-MDT0000 [11022.987094] Lustre: server umount lustre-MDT0000 complete [11024.868182] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11024.876916] LustreError: 348422:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11024.878772] Lustre: Skipped 2 previous similar messages [11024.906630] LustreError: 348422:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [11040.224823] Lustre: 340905:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790721562/real 1790721562] req@000000000b9c727e x1877707323437952/t0(0) o400->MGC192.168.204.126@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790721578 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11040.247240] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11042.958840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11050.474577] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xee594c2024f7c286 [11050.482509] Lustre: MGC192.168.204.126@tcp: Connection restored to 0@lo (at 0@lo) [11050.492620] Lustre: Skipped 2 previous similar messages [11050.806483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11050.863596] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11055.937222] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11055.945079] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11056.121922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11056.184672] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11056.236655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11067.062863] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11069.028762] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11081.696047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11081.699079] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11081.700765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11081.749514] Lustre: Skipped 3 previous similar messages [11083.139674] Lustre: server umount lustre-MDT0000 complete [11086.823701] LustreError: 348422:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11086.847356] LustreError: 348422:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 23 previous similar messages [11087.368472] LustreError: 347398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790721625 with bad export cookie 17174842354896978566 [11087.373941] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11087.379785] LustreError: 347398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11087.697291] Lustre: server umount lustre-MDT0001 complete [11102.289943] Lustre: server umount lustre-OST0000 complete [11121.031981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11121.578931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11126.427701] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11136.393511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11142.648236] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11151.995170] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11158.500400] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11165.950284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11167.408853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11173.017739] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11181.558451] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11187.173687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11187.202325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11187.219146] Lustre: Skipped 2 previous similar messages [11188.891755] Lustre: server umount lustre-OST0000 complete [11193.327176] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11199.575376] Lustre: server umount lustre-MDT0000 complete [11202.538412] LustreError: 353761:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11202.565947] LustreError: 353761:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11203.807466] LustreError: 353744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790721741 with bad export cookie 17174842354896979315 [11203.821194] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11203.825108] LustreError: 353744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11204.275282] Lustre: server umount lustre-MDT0001 complete [11220.936574] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [11225.283936] Key type lgssc unregistered [11225.622743] LNet: 357297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11225.643768] LNetError: 357297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11225.674684] LNet: Removed LNI 192.168.204.126@tcp [11227.019655] Key type .llcrypt unregistered [11227.022069] Key type ._llcrypt unregistered [11259.311506] Key type ._llcrypt registered [11259.316114] Key type .llcrypt registered [11259.486867] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [11260.844760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11261.178343] alg: No test for adler32 (adler32-zlib) [11262.439177] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [11262.798892] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [11264.519517] Key type lgssc registered [11265.902594] Lustre: Echo OBD driver; http://www.lustre.org/ [11273.931501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11273.951198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11275.317570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11279.492572] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11285.477335] LustreError: 358829:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11285.494169] LustreError: 358829:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11289.344258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11289.791295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11294.469966] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11304.292457] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11310.390202] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11317.789616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11318.176869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11320.256972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11324.971122] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11332.755161] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11346.400376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11346.407922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11346.422447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11347.518975] Lustre: server umount lustre-MDT0000 complete [11349.990839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11349.996295] LustreError: 358830:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11350.012717] Lustre: Skipped 1 previous similar message [11350.033821] LustreError: 358830:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11351.533375] LustreError: 358811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790721889 with bad export cookie 8359203816202927948 [11351.553814] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11352.037087] Lustre: server umount lustre-MDT0001 complete [11367.575657] Lustre: server umount lustre-OST0000 complete [11384.964246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11385.382358] LustreError: 362557:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11385.400050] LustreError: 362557:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11385.478721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11390.960872] LustreError: 362558:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11390.979903] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11395.126635] LustreError: 362557:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11401.509034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11402.137919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11408.341451] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11417.412713] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11424.388851] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11432.433143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11432.832348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11439.657211] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11448.710519] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11455.350945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11473.888799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11473.908205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11473.924658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11475.947795] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11475.948499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11475.988748] Lustre: Skipped 2 previous similar messages [11479.584512] Lustre: server umount lustre-MDT0000 complete [11481.064039] LustreError: 362554:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11481.078762] LustreError: 362554:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11483.623185] LustreError: 362538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790722021 with bad export cookie 8359203816202929068 [11483.629308] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11483.635395] LustreError: 362538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11483.941113] Lustre: server umount lustre-MDT0001 complete [11499.030791] Lustre: server umount lustre-OST0000 complete [11516.052277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11516.525168] LustreError: 366493:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11516.620968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11521.313876] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11530.351728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11535.520799] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11544.183219] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11550.449767] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11559.516036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11559.995444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11560.002693] Lustre: Skipped 1 previous similar message [11562.060308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11567.207726] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11576.194631] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11582.434883] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11582.449580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11582.460647] Lustre: Skipped 1 previous similar message [11582.465065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11586.016574] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11587.529080] Lustre: server umount lustre-OST0000 complete [11592.160468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11592.176478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11592.187026] Lustre: Skipped 1 previous similar message [11592.202271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11592.207798] Lustre: Skipped 2 previous similar messages [11597.926132] Lustre: server umount lustre-MDT0000 complete [11601.379146] LustreError: 367232:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11601.408579] LustreError: 367232:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11601.819685] LustreError: 366474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790722139 with bad export cookie 8359203816202930090 [11601.824819] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11601.837100] LustreError: 366474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11602.312905] Lustre: server umount lustre-MDT0001 complete [11615.690710] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [11619.079403] Key type lgssc unregistered [11619.429928] LNet: 370027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11619.450619] LNetError: 370027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11619.481519] LNet: Removed LNI 192.168.204.126@tcp [11620.517504] Key type .llcrypt unregistered [11620.523446] Key type ._llcrypt unregistered [11657.041854] Key type ._llcrypt registered [11657.043491] Key type .llcrypt registered [11657.185407] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [11658.115199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11658.208715] alg: No test for adler32 (adler32-zlib) [11659.374797] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [11659.702538] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [11661.456456] Key type lgssc registered [11662.538990] Lustre: Echo OBD driver; http://www.lustre.org/ [11669.603412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11669.639629] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11669.646803] LustreError: 371489:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11669.665451] LustreError: 371489:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11669.673402] LustreError: 371489:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11669.678943] LustreError: 371489:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11669.685592] LustreError: 371489:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11678.762077] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11678.768763] LustreError: 371888:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11678.783268] LustreError: 371888:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11678.791717] LustreError: 371888:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11678.800249] LustreError: 371888:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11678.806331] LustreError: 371888:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11701.303632] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [11704.609246] Key type lgssc unregistered [11704.951553] LNet: 372890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11704.962922] LNetError: 372890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11704.986872] LNet: Removed LNI 192.168.204.126@tcp [11705.963535] Key type .llcrypt unregistered [11705.967149] Key type ._llcrypt unregistered [11725.269692] Key type ._llcrypt registered [11725.277703] Key type .llcrypt registered [11725.381202] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [11744.724629] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [11746.191961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11746.302522] alg: No test for adler32 (adler32-zlib) [11747.401419] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [11747.626912] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [11749.345373] Key type lgssc registered [11750.712635] Lustre: Echo OBD driver; http://www.lustre.org/ [11791.768116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11791.823397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11793.259152] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11793.307193] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11793.396942] Lustre: lustre-MDT0000: new disk, initializing [11793.483879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11793.497282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11799.551615] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11812.829569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11813.040802] Lustre: 376063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11813.140239] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11813.153911] Lustre: Skipped 1 previous similar message [11813.260221] Lustre: lustre-MDT0001: new disk, initializing [11813.382332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11813.448798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11813.464628] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11818.831978] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11830.039236] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11835.971643] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11843.630838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11843.997497] Lustre: lustre-OST0000: new disk, initializing [11844.015644] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11844.021346] Lustre: 378033:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11844.153158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11851.184689] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11851.887138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11851.900346] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11852.028502] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11862.261417] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11867.793559] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11868.003527] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11873.719094] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11874.021721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11877.868933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11877.870819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11877.878226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11877.886464] LustreError: Skipped 1 previous similar message [11877.913457] Lustre: Skipped 1 previous similar message [11882.632484] Lustre: server umount lustre-OST0000 complete [11888.098746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11888.117068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11888.119566] Lustre: Skipped 1 previous similar message [11893.157723] Lustre: server umount lustre-MDT0000 complete [11896.816459] LustreError: 376860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790722434 with bad export cookie 13317241455103757118 [11896.817633] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11896.828735] LustreError: 376860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11897.222584] Lustre: server umount lustre-MDT0001 complete [11906.077651] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11907.741438] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 18:54:05 (1790722445) [11914.970191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11915.544247] LustreError: 380494:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11915.560703] LustreError: 380494:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11915.642133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11920.869421] LustreError: 380495:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11920.894313] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11925.985513] LustreError: 380494:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11930.432943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11930.822781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11935.711394] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11944.208948] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11950.987991] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11958.505391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11958.914549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11965.328253] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11973.330571] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11980.838749] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11981.090427] Lustre: lustre-OST0001: new disk, initializing [11981.093851] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11981.100465] Lustre: 383329:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11981.211648] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11987.560858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11987.576992] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11987.705720] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11988.276449] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11998.715343] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12003.304366] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12003.325589] Lustre: Skipped 1 previous similar message [12003.336908] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12003.339713] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12005.754165] Lustre: lustre-OST0001: Not available for connect from 192.168.204.26@tcp (stopping) [12005.761613] Lustre: Skipped 1 previous similar message [12008.855569] Lustre: server umount lustre-OST0001 complete [12010.833486] LustreError: 382447:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12018.696805] LustreError: 382316:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12018.713691] LustreError: 382316:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12026.848558] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12026.859800] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12026.876996] Lustre: Skipped 1 previous similar message [12026.889310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12026.902070] Lustre: Skipped 2 previous similar messages [12028.899257] LustreError: 382315:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12028.912912] LustreError: 382315:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12032.557260] Lustre: server umount lustre-OST0000 complete [12038.624323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12038.635924] LustreError: Skipped 1 previous similar message [12038.650299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12038.669881] Lustre: Skipped 1 previous similar message [12038.678437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12038.687539] Lustre: Skipped 3 previous similar messages [12043.076699] Lustre: server umount lustre-MDT0000 complete [12047.180578] LustreError: 383339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790722585 with bad export cookie 13317241455103757993 [12047.183595] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12047.196297] LustreError: 383339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12053.764091] Lustre: server umount lustre-MDT0001 complete [12067.201088] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12070.775705] Key type lgssc unregistered [12071.149902] LNet: 385315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12071.178656] LNetError: 385315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12071.211801] LNet: Removed LNI 192.168.204.126@tcp [12072.481172] Key type .llcrypt unregistered [12072.485546] Key type ._llcrypt unregistered [12131.065719] Key type ._llcrypt registered [12131.070127] Key type .llcrypt registered [12131.187500] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [12132.700356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12132.801616] alg: No test for adler32 (adler32-zlib) [12133.815406] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [12134.096893] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [12135.775303] Key type lgssc registered [12137.043895] Lustre: Echo OBD driver; http://www.lustre.org/ [12146.140594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12146.193502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12147.631466] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12147.651953] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12147.918748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12152.729750] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12162.037356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12162.257476] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12162.314823] Lustre: 388137:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12162.624298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12167.670865] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12176.926091] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12184.905517] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12188.138652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12188.152747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12188.179419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12189.664544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12189.666210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12193.937942] Lustre: server umount lustre-MDT0000 complete [12198.318132] LustreError: 388130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790722736 with bad export cookie 13208226760583568213 [12198.334199] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12198.341114] LustreError: 388130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12198.844578] Lustre: server umount lustre-MDT0001 complete [12209.764915] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 18:59:07 (1790722747) [12218.460724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12218.948683] LustreError: 390653:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12218.976715] LustreError: 390653:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12219.055859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12224.186268] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12224.485321] LustreError: 390654:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12224.506408] LustreError: 390654:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12229.610073] LustreError: 390653:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12234.661926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12234.724778] LustreError: 390654:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12235.121258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12240.545980] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12248.699308] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12254.895817] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12261.974909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12262.190650] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12262.364309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12268.785282] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12277.067488] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12286.453880] Lustre: Permanently deactivating lustre-OST0000 [12286.474922] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12296.225878] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12297.241701] LustreError: 393642:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12297.256668] LustreError: 393642:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12297.263734] LustreError: 393642:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12301.229889] LustreError: 393691:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12301.240413] LustreError: 393691:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12309.872436] Lustre: server umount lustre-OST0000 complete [12313.227937] LustreError: 393941:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12313.236309] LustreError: 393941:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12317.153967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12317.162339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12317.192698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12317.202672] Lustre: Skipped 1 previous similar message [12322.785131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12322.791393] Lustre: Skipped 2 previous similar messages [12324.962184] Lustre: server umount lustre-MDT0000 complete [12327.913250] LustreError: 390653:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12327.956347] LustreError: 390653:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12330.515625] LustreError: 390634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790722868 with bad export cookie 13208226760583568801 [12330.528104] LustreError: 390634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12330.531863] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12341.029687] Lustre: server umount lustre-MDT0001 complete [12353.631919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12359.119651] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12363.276877] LustreError: 395111:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12371.000753] Lustre: 395450:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.126@tcp [12373.661644] Lustre: server umount MGS complete [12387.260163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12387.709757] LustreError: 396175:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12387.733041] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12387.737430] Lustre: Skipped 1 previous similar message [12387.810350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12392.563612] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12402.169655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12402.606820] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12407.472832] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12416.376722] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12423.734831] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12426.052980] Lustre: Permanently reactivating lustre-OST0000 [12426.064700] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12426.069814] Lustre: Skipped 2 previous similar messages [12452.319122] LustreError: 397909:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12461.433192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12461.905719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12461.927783] Lustre: Skipped 1 previous similar message [12469.601126] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12479.294197] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12487.402915] Lustre: server umount lustre-OST0000 complete [12493.793888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12493.809788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12493.823790] Lustre: Skipped 2 previous similar messages [12493.833920] Lustre: Skipped 1 previous similar message [12504.031300] Lustre: 387449:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790723025/real 1790723025] req@000000004995f562 x1877708816615296/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790723041 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12504.033965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12504.080490] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12504.088712] Lustre: Skipped 3 previous similar messages [12504.928369] Lustre: 387452:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790723026/real 1790723026] req@00000000d83eb659 x1877708816616064/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790723042 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12506.079152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12506.219277] Lustre: server umount lustre-MDT0000 complete [12509.152585] Lustre: 387450:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790723031/real 1790723031] req@00000000afc58857 x1877708816617856/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790723047 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12509.159053] LustreError: 396176:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12509.202428] LustreError: 396176:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12510.021454] LustreError: 396155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790723047 with bad export cookie 13208226760583569851 [12510.031337] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12510.034835] LustreError: 396155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12510.462240] Lustre: server umount lustre-MDT0001 complete [12522.431519] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12526.110516] Key type lgssc unregistered [12526.461637] LNet: 400492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12526.469348] LNetError: 400492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12527.528714] LNet: Removed LNI 192.168.204.126@tcp [12528.575810] Key type .llcrypt unregistered [12528.586651] Key type ._llcrypt unregistered [12548.632412] Key type ._llcrypt registered [12548.638092] Key type .llcrypt registered [12548.790973] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [12564.883352] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [12565.976215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12566.030064] alg: No test for adler32 (adler32-zlib) [12567.345671] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [12567.587650] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [12569.335227] Key type lgssc registered [12570.534858] Lustre: Echo OBD driver; http://www.lustre.org/ [12608.676951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12608.757370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12610.108903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12610.145774] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12610.269848] Lustre: lustre-MDT0000: new disk, initializing [12610.404148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12610.450545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12615.225965] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12626.799845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12626.964600] Lustre: 403680:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12627.078600] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12627.083269] Lustre: Skipped 1 previous similar message [12627.180890] Lustre: lustre-MDT0001: new disk, initializing [12627.277594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12627.307882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12627.327835] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12632.493943] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12643.515820] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12650.130712] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12657.553840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12657.896404] Lustre: lustre-OST0000: new disk, initializing [12657.901623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12657.909865] Lustre: 405648:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12658.016795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12664.579854] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12665.977745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12666.004542] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12666.180250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12675.121233] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12679.977606] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12680.169596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12684.767219] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12684.979350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12691.938897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12691.959364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12691.964761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12691.990355] Lustre: Skipped 1 previous similar message [12692.005812] Lustre: Skipped 1 previous similar message [12697.061626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12697.075620] Lustre: Skipped 1 previous similar message [12701.151571] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12701.379837] Lustre: server umount lustre-OST0000 complete [12707.315638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12707.352724] Lustre: Skipped 1 previous similar message [12707.354062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12707.381209] Lustre: Skipped 1 previous similar message [12712.383773] Lustre: server umount lustre-MDT0000 complete [12716.190221] LustreError: 404484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790723254 with bad export cookie 7442000361370034070 [12716.208230] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12716.209349] LustreError: 404484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12716.622648] Lustre: server umount lustre-MDT0001 complete [12726.934522] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12728.855594] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 19:07:46 (1790723266) [12736.669885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12737.290844] LustreError: 408111:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12737.314060] LustreError: 408111:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12737.389075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12742.634541] LustreError: 408112:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12742.744105] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12747.749245] LustreError: 408111:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12752.174456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12752.507882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12757.795529] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12765.647336] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12771.358557] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12777.998738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12778.414614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12784.267248] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12792.548951] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12794.766917] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12797.580252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12808.162134] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12808.172984] LustreError: Skipped 1 previous similar message [12808.180310] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12808.201065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12813.287735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12813.294775] Lustre: Skipped 1 previous similar message [12813.735972] Lustre: server umount lustre-OST0000 complete [12818.407967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12818.417434] Lustre: Skipped 1 previous similar message [12823.520856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12823.531624] Lustre: Skipped 4 previous similar messages [12823.851477] Lustre: server umount lustre-MDT0000 complete [12827.909783] LustreError: 408091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790723365 with bad export cookie 7442000361370034952 [12827.917232] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12827.925696] LustreError: 408091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12828.358580] Lustre: server umount lustre-MDT0001 complete [12840.447896] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12843.753771] Key type lgssc unregistered [12844.108595] LNet: 411834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12844.126660] LNetError: 411834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12844.144163] LNet: Removed LNI 192.168.204.126@tcp [12845.175857] Key type .llcrypt unregistered [12845.178130] Key type ._llcrypt unregistered [12869.867532] Key type ._llcrypt registered [12869.869379] Key type .llcrypt registered [12869.969372] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12870.236432] Key type .llcrypt unregistered [12870.238778] Key type ._llcrypt unregistered [12889.557941] Key type ._llcrypt registered [12889.561921] Key type .llcrypt registered [12889.678565] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [12890.687327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12890.967956] alg: No test for adler32 (adler32-zlib) [12892.224971] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [12892.524740] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [12894.207244] Key type lgssc registered [12895.896728] Lustre: Echo OBD driver; http://www.lustre.org/ [12904.455972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12904.493671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12906.038980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12911.429489] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12915.178016] LustreError: 414063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12915.214976] LustreError: 414063:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12920.289604] LustreError: 414064:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12922.691819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12923.100449] LustreError: 414063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12923.158623] LustreError: 414063:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12923.161310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12928.485485] LustreError: 414064:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12928.508025] LustreError: 414064:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12930.419339] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12930.558950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12935.649855] LustreError: 414063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12935.659612] LustreError: 414063:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12938.976622] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12945.908933] LustreError: 414059:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12945.944698] LustreError: 414059:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12947.657786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12948.243332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12953.175328] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12963.810167] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12963.815981] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12963.841189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12963.864438] Lustre: Skipped 1 previous similar message [12969.009527] Lustre: server umount lustre-OST0000 complete [12974.445519] Lustre: server umount lustre-MDT0000 complete [12978.940156] LustreError: 414046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790723516 with bad export cookie 18314825016907427259 [12978.964727] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12981.511617] Lustre: server umount lustre-MDT0001 complete [12995.225493] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12999.079612] Key type lgssc unregistered [12999.572782] LNet: 417211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12999.590606] LNetError: 417211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12999.615906] LNet: Removed LNI 192.168.204.126@tcp [13000.750375] Key type .llcrypt unregistered [13000.752520] Key type ._llcrypt unregistered [13027.135844] Key type ._llcrypt registered [13027.142850] Key type .llcrypt registered [13027.262518] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [13028.215102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13028.332988] alg: No test for adler32 (adler32-zlib) [13029.426746] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [13029.621857] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [13031.287231] Key type lgssc registered [13032.499274] Lustre: Echo OBD driver; http://www.lustre.org/ [13041.311541] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13041.331636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13056.929641] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13062.047696] LustreError: 418619:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.126@tcp: failed processing log, type 4: rc = -110 [13088.799184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13096.545726] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13106.073716] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13114.384270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13114.956976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13115.016816] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13119.835702] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13119.971864] LustreError: 418645:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13119.997502] LustreError: 418645:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13124.078072] LustreError: 419656:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13124.098669] LustreError: 419656:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13128.886335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13129.265139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13134.115287] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13144.328370] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13151.571366] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13165.026033] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13165.030527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13165.043382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13165.051880] Lustre: Skipped 1 previous similar message [13165.064366] Lustre: Skipped 1 previous similar message [13167.253788] Lustre: server umount lustre-OST0000 complete [13173.690457] Lustre: server umount lustre-MDT0000 complete [13175.264277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13175.266064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13175.292237] Lustre: Skipped 1 previous similar message [13175.301350] LustreError: 419662:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13178.006591] LustreError: 418626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790723715 with bad export cookie 13994297267900860592 [13178.008615] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13178.020707] LustreError: 418626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13178.562397] Lustre: server umount lustre-MDT0001 complete [13193.944960] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [13197.442820] Key type lgssc unregistered [13197.812989] LNet: 422191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13197.824889] LNetError: 422191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13197.840287] LNet: Removed LNI 192.168.204.126@tcp [13198.635417] Key type .llcrypt unregistered [13198.637202] Key type ._llcrypt unregistered [13224.723615] Key type ._llcrypt registered [13224.735331] Key type .llcrypt registered [13224.866146] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [13226.050198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13226.135332] alg: No test for adler32 (adler32-zlib) [13227.440678] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [13227.745464] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [13229.423398] Key type lgssc registered [13231.057016] Lustre: Echo OBD driver; http://www.lustre.org/ [13238.679424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13238.718346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13240.243108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13245.516531] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13250.534881] LustreError: 423622:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13250.559714] LustreError: 423622:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13255.650860] LustreError: 423623:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13256.382554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13256.945699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13262.278756] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13270.928739] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13278.203943] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13286.695793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13287.146774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13293.928468] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13302.116470] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13310.699734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13318.624263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13318.635690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13318.665027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13321.188937] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13321.193760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13321.217588] Lustre: Skipped 1 previous similar message [13321.230555] Lustre: Skipped 3 previous similar messages [13323.381359] Lustre: server umount lustre-MDT0000 complete [13324.105615] LustreError: 425223:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13326.308379] LustreError: 423618:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13326.325651] LustreError: 423618:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13326.371074] LustreError: 423601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.26@tcp arrived at 1790723864 with bad export cookie 4964891983365000171 [13328.251745] LustreError: 423603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790723866 with bad export cookie 4964891983364999506 [13328.268808] LustreError: 423603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13328.276973] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13331.425352] LustreError: 423619:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13331.426522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13331.454445] LustreError: 423619:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13331.472626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13333.481643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13334.648308] Lustre: server umount lustre-OST0000 complete [13343.005689] Lustre: server umount lustre-MDT0001 complete [13355.480033] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [13359.377626] Key type lgssc unregistered [13359.966386] LNet: 427327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13360.004487] LNetError: 427327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13360.072298] LNet: Removed LNI 192.168.204.126@tcp [13361.092711] Key type .llcrypt unregistered [13361.096842] Key type ._llcrypt unregistered [13391.657854] Key type ._llcrypt registered [13391.661689] Key type .llcrypt registered [13391.768683] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [13393.141314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13393.166310] alg: No test for adler32 (adler32-zlib) [13394.356612] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [13394.754625] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [13396.432552] Key type lgssc registered [13397.889160] Lustre: Echo OBD driver; http://www.lustre.org/ [13406.359274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13406.394701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13408.017172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13412.948352] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13417.444767] LustreError: 428764:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13417.477342] LustreError: 428764:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13421.770586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13422.130676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13426.999244] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13436.219754] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13443.279870] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13451.342758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13451.790806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13452.867693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13459.046523] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13467.976379] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13473.249322] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13473.257357] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13473.269995] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13476.204292] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [13477.345198] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13477.347806] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13479.627850] Lustre: server umount lustre-MDT0001 complete [13481.289548] LustreError: 428758:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13482.471826] LustreError: 430204:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13482.501203] LustreError: 430204:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13487.584308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13487.586580] LustreError: 428763:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13487.603356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13487.618786] Lustre: Skipped 1 previous similar message [13492.694540] Lustre: server umount lustre-OST0000 complete [13492.719786] LustreError: 430204:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13497.768968] Lustre: server umount lustre-MDT0000 complete [13513.862131] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [13517.113495] Key type lgssc unregistered [13517.363710] LNet: 432448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13517.370415] LNetError: 432448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13517.395855] LNet: Removed LNI 192.168.204.126@tcp [13518.411142] Key type .llcrypt unregistered [13518.413503] Key type ._llcrypt unregistered [13544.138863] Key type ._llcrypt registered [13544.149680] Key type .llcrypt registered [13544.273727] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [13559.252260] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [13560.370599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13560.448516] alg: No test for adler32 (adler32-zlib) [13561.492336] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [13561.752384] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [13563.423221] Key type lgssc registered [13564.482718] Lustre: Echo OBD driver; http://www.lustre.org/ [13601.457487] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [13618.610561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13618.655826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13619.981658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13620.031981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13620.218644] Lustre: lustre-MDT0000: new disk, initializing [13620.365089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13620.395721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13626.349957] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13646.339868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13646.772538] Lustre: lustre-OST0000: new disk, initializing [13646.786984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13646.800930] Lustre: Skipped 1 previous similar message [13646.808575] Lustre: 437500:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13646.921632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13653.647247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13653.658259] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13653.694436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13654.661172] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13665.988573] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13683.300162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13683.426637] Lustre: 436482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13683.546602] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13683.656496] Lustre: lustre-MDT0001: new disk, initializing [13683.821621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13683.890525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13683.908298] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13688.891606] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13698.685978] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13698.949265] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13703.968895] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13704.347983] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13709.163537] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [13709.795410] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13709.801976] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13709.827720] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13709.854632] Lustre: Skipped 1 previous similar message [13714.250521] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [13714.765626] Lustre: server umount lustre-MDT0001 complete [13720.051895] LustreError: 436488:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13720.073898] LustreError: 436488:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [13725.158800] LustreError: 436493:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13725.179464] LustreError: 436493:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13730.275325] LustreError: 436489:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13730.292037] LustreError: 436489:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13735.394702] LustreError: 436494:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13735.396322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13735.421802] Lustre: Skipped 1 previous similar message [13735.426301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13740.499981] Lustre: server umount lustre-OST0000 complete [13740.523770] LustreError: 436493:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13751.377378] Lustre: server umount lustre-MDT0000 complete [13766.713662] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [13769.979993] Key type lgssc unregistered [13770.431381] LNet: 441251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13770.443633] LNetError: 441251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13770.483231] LNet: Removed LNI 192.168.204.126@tcp [13771.802904] Key type .llcrypt unregistered [13771.805564] Key type ._llcrypt unregistered [13816.482236] Key type ._llcrypt registered [13816.486565] Key type .llcrypt registered [13816.639040] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [13816.901225] Key type .llcrypt unregistered [13816.904114] Key type ._llcrypt unregistered [13834.486261] Key type ._llcrypt registered [13834.491270] Key type .llcrypt registered [13834.668529] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [13836.107485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13836.129055] alg: No test for adler32 (adler32-zlib) [13837.297613] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [13837.532540] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [13839.271249] Key type lgssc registered [13841.070934] Lustre: Echo OBD driver; http://www.lustre.org/ [13872.981844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13873.009020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13874.308749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13874.355851] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13874.461992] Lustre: lustre-MDT0000: new disk, initializing [13874.564985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13874.591113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13879.441529] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13890.279260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13890.422616] Lustre: 444645:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13890.483707] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13890.488031] Lustre: Skipped 1 previous similar message [13890.549920] Lustre: lustre-MDT0001: new disk, initializing [13890.644699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13890.682209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13890.696665] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13895.484191] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13906.432728] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13912.469900] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13919.875539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13920.216603] Lustre: lustre-OST0000: new disk, initializing [13920.223939] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13920.229374] Lustre: 446613:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13920.331150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13926.764512] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13930.579322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13930.590484] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13930.707287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13936.841704] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13946.341061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13946.344937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13946.363590] Lustre: Skipped 2 previous similar messages [13946.379862] Lustre: Skipped 2 previous similar messages [13949.454626] Lustre: server umount lustre-MDT0000 complete [13953.928302] LustreError: 446612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790724491 with bad export cookie 831909767026201877 [13953.933391] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13953.950584] LustreError: 446612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13954.466767] Lustre: server umount lustre-MDT0001 complete [13969.665178] Lustre: server umount lustre-OST0000 complete [13980.547806] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 19:28:38 (1790724518) [13987.690626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13988.031937] LustreError: 448641:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13988.047188] LustreError: 448641:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13988.124663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13992.733518] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13993.440208] LustreError: 448642:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13998.566891] LustreError: 448641:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14003.685881] LustreError: 448642:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14004.219386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14004.824135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14010.834847] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [14021.966634] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14029.916669] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14051.373849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14051.667469] Lustre: Found index 0 for lustre-OST0000, updating log [14051.886664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14058.593577] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [14059.077689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14067.137485] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14074.331649] Lustre: server umount lustre-OST0000 complete [14090.658766] Lustre: 443165:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790724612/real 1790724612] req@000000004882c18a x1877710602313088/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790724628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14090.689977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14091.744500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14091.748351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14091.757429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14092.780848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14092.789868] Lustre: Skipped 1 previous similar message [14097.169673] Lustre: server umount lustre-MDT0000 complete [14098.913881] LustreError: 449567:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14098.933715] LustreError: 449567:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14101.566598] LustreError: 450488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790724639 with bad export cookie 831909767026203270 [14101.568787] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14101.582885] LustreError: 450488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14101.947793] Lustre: server umount lustre-MDT0001 complete [14112.651413] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 19:30:49 (1790724649) [14114.207255] LNet: 1 local NIs in recovery (showing 1): 192.168.204.126@tcp [14733.956798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14734.412178] LustreError: 455053:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14734.493287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14738.640035] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [14739.938592] LustreError: 455054:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14745.061215] LustreError: 455053:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14747.828503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14748.333556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14752.984769] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [14761.790337] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14768.662680] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14788.602275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14788.932168] Lustre: Found index 0 for lustre-OST0000, updating log [14789.354169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14792.095240] LNet: 1 local NIs in recovery (showing 1): 192.168.204.126@tcp [14795.981326] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [14803.892955] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14807.009308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14807.024152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14807.032341] Lustre: Skipped 1 previous similar message [14807.040707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14807.046834] Lustre: Skipped 2 previous similar messages [14812.774363] Lustre: server umount lustre-OST0000 complete [14817.328099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14817.338200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14817.352432] Lustre: Skipped 1 previous similar message [14817.370738] Lustre: Skipped 1 previous similar message [14818.402231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14818.423198] Lustre: Skipped 1 previous similar message [14820.458238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14820.478032] Lustre: Skipped 3 previous similar messages [14823.797628] Lustre: server umount lustre-MDT0000 complete [14824.610100] LustreError: 455054:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14824.624202] LustreError: 455054:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [14827.844599] LustreError: 456908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790725365 with bad export cookie 831909767026204341 [14827.856205] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14827.868187] LustreError: 456908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14828.185536] Lustre: server umount lustre-MDT0001 complete [14852.533055] LNet: 1 local NIs in recovery (showing 1): 192.168.204.126@tcp [14860.012714] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [14872.579609] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [14913.581633] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 19:44:11 (1790725451) [14934.822708] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [14938.296675] Key type lgssc unregistered [14938.665284] LNet: 461671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14938.688756] LNetError: 461671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15050.345465] LNet: Removed LNI 192.168.204.126@tcp [15051.446111] Key type .llcrypt unregistered [15051.451496] Key type ._llcrypt unregistered [15075.404805] Key type ._llcrypt registered [15075.407181] Key type .llcrypt registered [15075.522574] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [15076.744929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15077.005348] alg: No test for adler32 (adler32-zlib) [15078.179330] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [15078.439356] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [15080.159206] Key type lgssc registered [15081.614536] Lustre: Echo OBD driver; http://www.lustre.org/ [15087.024907] LNet: Added LNI 192.168.204.126@tcp100 [8/256/0/180] [15087.999400] LNet: Added LNI 192.168.204.126@tcp101 [8/256/0/180] [15089.008576] LNet: Added LNI 192.168.204.126@tcp102 [8/256/0/180] [15089.952894] LNet: Added LNI 192.168.204.126@tcp103 [8/256/0/180] [15090.860088] LNet: Added LNI 192.168.204.126@tcp104 [8/256/0/180] [15091.845756] LNet: Added LNI 192.168.204.126@tcp105 [8/256/0/180] [15092.803916] LNet: Added LNI 192.168.204.126@tcp106 [8/256/0/180] [15093.942947] LNet: Added LNI 192.168.204.126@tcp107 [8/256/0/180] [15095.265252] LNet: Added LNI 192.168.204.126@tcp108 [8/256/0/180] [15096.498863] LNet: Added LNI 192.168.204.126@tcp109 [8/256/0/180] [15097.741968] LNet: Added LNI 192.168.204.126@tcp110 [8/256/0/180] [15098.757063] LNet: Added LNI 192.168.204.126@tcp111 [8/256/0/180] [15099.826727] LNet: Added LNI 192.168.204.126@tcp112 [8/256/0/180] [15100.837128] LNet: Added LNI 192.168.204.126@tcp113 [8/256/0/180] [15101.943927] LNet: Added LNI 192.168.204.126@tcp114 [8/256/0/180] [15103.004085] LNet: Added LNI 192.168.204.126@tcp115 [8/256/0/180] [15104.039098] LNet: Added LNI 192.168.204.126@tcp116 [8/256/0/180] [15105.149702] LNet: Added LNI 192.168.204.126@tcp117 [8/256/0/180] [15106.298334] LNet: Added LNI 192.168.204.126@tcp118 [8/256/0/180] [15107.336470] LNet: Added LNI 192.168.204.126@tcp119 [8/256/0/180] [15108.247681] LNet: Added LNI 192.168.204.126@tcp120 [8/256/0/180] [15109.429930] LNet: Added LNI 192.168.204.126@tcp121 [8/256/0/180] [15110.495363] LNet: Added LNI 192.168.204.126@tcp122 [8/256/0/180] [15111.541942] LNet: Added LNI 192.168.204.126@tcp123 [8/256/0/180] [15112.575420] LNet: Added LNI 192.168.204.126@tcp124 [8/256/0/180] [15113.622735] LNet: Added LNI 192.168.204.126@tcp125 [8/256/0/180] [15114.565023] LNet: Added LNI 192.168.204.126@tcp126 [8/256/0/180] [15115.421804] LNet: Added LNI 192.168.204.126@tcp127 [8/256/0/180] [15116.531292] LNet: Added LNI 192.168.204.126@tcp128 [8/256/0/180] [15117.517723] LNet: Added LNI 192.168.204.126@tcp129 [8/256/0/180] [15118.695908] LNet: Added LNI 192.168.204.126@tcp130 [8/256/0/180] [15119.646800] LNet: Added LNI 192.168.204.126@tcp131 [8/256/0/180] [15120.549438] LNet: Added LNI 192.168.204.126@tcp132 [8/256/0/180] [15121.593192] LNet: Added LNI 192.168.204.126@tcp133 [8/256/0/180] [15122.471710] LNet: Added LNI 192.168.204.126@tcp134 [8/256/0/180] [15123.461196] LNet: Added LNI 192.168.204.126@tcp135 [8/256/0/180] [15131.553368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [15131.571815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15132.753127] Lustre: 464719:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [15132.861305] LNetError: 464750:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [15132.908580] LNetError: Protocol error connecting to 192.168.204.126@tcp120 at host 192.168.204.126:988: is it running a compatible version of Lustre? [15132.923567] LNetError: Skipped 4 previous similar messages [15133.020225] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15133.052181] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15133.127079] Lustre: lustre-MDT0000: new disk, initializing [15133.263503] Lustre: 464777:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [15133.281129] Lustre: 464777:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15133.304811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15133.335774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15133.729654] LNet: 5 local NIs in recovery (showing 5): 192.168.204.126@tcp126, 192.168.204.126@tcp127, 192.168.204.126@tcp128, 192.168.204.126@tcp129, 192.168.204.126@tcp120 [15133.731996] LNetError: 464750:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [15133.770718] LNetError: 464750:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [15133.789855] LNetError: Protocol error connecting to 192.168.204.126@tcp100 at host 192.168.204.126:988: is it running a compatible version of Lustre? [15133.810960] LNetError: Skipped 2 previous similar messages [15138.438712] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [15150.111943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15150.426657] Lustre: 464738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15150.534294] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15150.551769] Lustre: Skipped 1 previous similar message [15150.695386] Lustre: lustre-MDT0001: new disk, initializing [15150.751826] Lustre: 465541:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [15150.761467] Lustre: 465541:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15150.990226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15151.079870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15151.119041] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15156.004561] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [15158.849928] Lustre: 465904:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [15158.861867] Lustre: 465904:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15167.005678] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15167.648405] LNetError: 464742:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [15167.658732] LNetError: Protocol error connecting to 192.168.204.126@tcp100 at host 192.168.204.126:988: is it running a compatible version of Lustre? [15169.702053] LNetError: 464747:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [15169.715145] LNetError: Protocol error connecting to 192.168.204.126@tcp124 at host 192.168.204.126:988: is it running a compatible version of Lustre? [15174.056761] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15174.824618] LNetError: 464749:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [15174.830028] LNetError: Protocol error connecting to 192.168.204.126@tcp125 at host 192.168.204.126:988: is it running a compatible version of Lustre? [15181.229657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15181.514842] Lustre: lustre-OST0000: new disk, initializing [15181.519100] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15181.526706] Lustre: 466714:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15181.638800] Lustre: 466719:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [15181.647670] Lustre: 466719:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15181.710249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15183.012313] LNetError: 464743:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [15183.037670] LNetError: 464743:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [15183.054242] LNetError: Protocol error connecting to 192.168.204.126@tcp124 at host 192.168.204.126:988: is it running a compatible version of Lustre? [15183.070941] LNetError: Skipped 1 previous similar message [15188.091777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15188.106285] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15188.293225] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15188.978283] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [15199.266555] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15200.423322] LNetError: 462170:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [15200.432155] LNetError: 462170:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [15200.438419] LNetError: Protocol error connecting to 192.168.204.126@tcp125 at host 192.168.204.126:988: is it running a compatible version of Lustre? [15200.448678] LNetError: Skipped 3 previous similar messages [15208.931477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15208.933636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15208.938097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15208.951628] Lustre: Skipped 1 previous similar message [15212.872483] Lustre: server umount lustre-OST0000 complete [15217.636949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15217.652542] LustreError: Skipped 1 previous similar message [15217.663352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15217.688234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15217.701309] Lustre: Skipped 1 previous similar message [15219.171100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15219.175368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15222.876297] Lustre: server umount lustre-MDT0000 complete [15226.294591] LustreError: 464728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790725764 with bad export cookie 5092147680365027545 [15226.298561] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15226.308822] LustreError: 464728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15226.707486] Lustre: server umount lustre-MDT0001 complete [15233.185611] LNetError: 464747:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [15233.200603] LNetError: 464747:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [15233.211109] LNetError: Protocol error connecting to 192.168.204.126@tcp124 at host 192.168.204.126:988: is it running a compatible version of Lustre? [15233.228392] LNetError: Skipped 3 previous similar messages [15240.929186] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [15244.272279] Key type lgssc unregistered [15244.662648] LNet: 468627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15244.699222] LNetError: 468627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15244.713547] LNet: Removed LNI 192.168.204.126@tcp [15245.727913] LNet: Removed LNI 192.168.204.126@tcp100 [15245.733403] LNetError: 468627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15245.746225] LNetError: 468627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15245.770473] LNet: Removed LNI 192.168.204.126@tcp101 [15246.815482] LNet: Removed LNI 192.168.204.126@tcp102 [15246.838391] LNetError: 468627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15246.859172] LNetError: 468627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15246.876728] LNet: Removed LNI 192.168.204.126@tcp103 [15247.904930] LNet: Removed LNI 192.168.204.126@tcp104 [15248.927372] LNet: Removed LNI 192.168.204.126@tcp105 [15248.931385] LNetError: 468627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15248.939204] LNetError: 468627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15248.947611] LNet: Removed LNI 192.168.204.126@tcp106 [15250.015621] LNet: Removed LNI 192.168.204.126@tcp107 [15251.047062] LNet: Removed LNI 192.168.204.126@tcp108 [15252.063161] LNet: Removed LNI 192.168.204.126@tcp109 [15253.087246] LNet: Removed LNI 192.168.204.126@tcp110 [15253.104853] LNetError: 468627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15253.129721] LNetError: 468627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15253.145491] LNet: Removed LNI 192.168.204.126@tcp111 [15254.176537] LNet: Removed LNI 192.168.204.126@tcp112 [15255.199329] LNet: Removed LNI 192.168.204.126@tcp113 [15256.223977] LNet: Removed LNI 192.168.204.126@tcp114 [15257.247227] LNet: Removed LNI 192.168.204.126@tcp115 [15258.271152] LNet: Removed LNI 192.168.204.126@tcp116 [15259.295168] LNet: Removed LNI 192.168.204.126@tcp117 [15260.320076] LNet: Removed LNI 192.168.204.126@tcp118 [15261.344251] LNet: Removed LNI 192.168.204.126@tcp119 [15261.353039] LNetError: 468627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15261.372533] LNetError: 468627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15262.431159] LNet: Removed LNI 192.168.204.126@tcp120 [15263.455161] LNet: Removed LNI 192.168.204.126@tcp121 [15264.479166] LNet: Removed LNI 192.168.204.126@tcp122 [15265.503172] LNet: Removed LNI 192.168.204.126@tcp123 [15266.528606] LNet: Removed LNI 192.168.204.126@tcp124 [15267.551647] LNet: Removed LNI 192.168.204.126@tcp125 [15268.575178] LNet: Removed LNI 192.168.204.126@tcp126 [15269.599966] LNet: Removed LNI 192.168.204.126@tcp127 [15270.623468] LNet: Removed LNI 192.168.204.126@tcp128 [15271.647590] LNet: Removed LNI 192.168.204.126@tcp129 [15271.649770] LNet: Removed LNI 192.168.204.126@tcp130 [15271.654848] LNet: Removed LNI 192.168.204.126@tcp131 [15271.667655] LNet: Removed LNI 192.168.204.126@tcp132 [15271.685519] LNet: Removed LNI 192.168.204.126@tcp133 [15271.687510] LNet: Removed LNI 192.168.204.126@tcp134 [15271.715923] LNet: Removed LNI 192.168.204.126@tcp135 [15272.622186] Key type .llcrypt unregistered [15272.624308] Key type ._llcrypt unregistered [15300.528960] Key type ._llcrypt registered [15300.530599] Key type .llcrypt registered [15300.628320] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [15320.136906] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [15321.810629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15321.866208] alg: No test for adler32 (adler32-zlib) [15323.048148] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [15323.339242] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [15325.024198] Key type lgssc registered [15326.350572] Lustre: Echo OBD driver; http://www.lustre.org/ [15369.681492] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 19:51:46 (1790725906) [15389.398918] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [15392.616869] Key type lgssc unregistered [15392.893727] LNet: 472734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15392.900996] LNetError: 472734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15392.924515] LNet: Removed LNI 192.168.204.126@tcp [15393.780926] Key type .llcrypt unregistered [15393.783287] Key type ._llcrypt unregistered [15414.537747] Key type ._llcrypt registered [15414.540321] Key type .llcrypt registered [15414.644818] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [15415.945978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15416.090936] alg: No test for adler32 (adler32-zlib) [15417.378591] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [15417.779490] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [15419.536638] Key type lgssc registered [15420.903822] Lustre: Echo OBD driver; http://www.lustre.org/ [15431.119783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [15431.157698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15432.417783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15432.452792] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15432.521514] Lustre: lustre-MDT0000: new disk, initializing [15432.608687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15432.630696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15437.406847] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [15448.490822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15448.664362] Lustre: 473963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15448.746812] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15448.754432] Lustre: Skipped 1 previous similar message [15448.870698] Lustre: lustre-MDT0001: new disk, initializing [15449.028117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15449.088166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15449.100366] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15453.899723] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [15464.117961] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15470.176465] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15478.044097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15478.378978] Lustre: lustre-OST0000: new disk, initializing [15478.382922] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15478.389529] Lustre: 475934:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15478.459223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15485.641315] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [15487.076760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15487.099069] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15487.254748] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15496.428318] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15500.537607] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing lnet_if_list [15507.286213] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15507.374935] LNet: Added LNI 192.168.204.126@tcp10 [8/256/0/180] [15514.354432] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15514.408688] LNet: Added LNI 192.168.204.126@tcp11 [8/256/0/180] [15519.684839] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15519.719163] LNet: Added LNI 192.168.204.126@tcp12 [8/256/0/180] [15524.683769] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15524.718041] LNet: Added LNI 192.168.204.126@tcp13 [8/256/0/180] [15530.080744] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15530.123449] LNet: Added LNI 192.168.204.126@tcp14 [8/256/0/180] [15534.804917] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15534.846775] LNet: Added LNI 192.168.204.126@tcp15 [8/256/0/180] [15541.161651] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15541.187424] LNet: Added LNI 192.168.204.126@tcp16 [8/256/0/180] [15548.104800] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15548.143562] LNet: Added LNI 192.168.204.126@tcp17 [8/256/0/180] [15554.049830] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15554.097961] LNet: Added LNI 192.168.204.126@tcp18 [8/256/0/180] [15560.383053] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15560.411354] LNet: Added LNI 192.168.204.126@tcp19 [8/256/0/180] [15567.180623] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15567.242343] LNet: Added LNI 192.168.204.126@tcp20 [8/256/0/180] [15573.359910] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15573.409672] LNet: Added LNI 192.168.204.126@tcp21 [8/256/0/180] [15580.106065] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15580.167172] LNet: Added LNI 192.168.204.126@tcp22 [8/256/0/180] [15587.521403] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15587.561058] LNet: Added LNI 192.168.204.126@tcp23 [8/256/0/180] [15595.016733] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15595.080092] LNet: Added LNI 192.168.204.126@tcp24 [8/256/0/180] [15602.063939] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15602.099893] LNet: Added LNI 192.168.204.126@tcp25 [8/256/0/180] [15609.714934] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15609.763743] LNet: Added LNI 192.168.204.126@tcp26 [8/256/0/180] [15617.686862] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15617.728905] LNet: Added LNI 192.168.204.126@tcp27 [8/256/0/180] [15624.961707] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15624.995807] LNet: Added LNI 192.168.204.126@tcp28 [8/256/0/180] [15632.342508] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15632.365303] LNet: Added LNI 192.168.204.126@tcp29 [8/256/0/180] [15638.997405] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15639.046251] LNet: Added LNI 192.168.204.126@tcp30 [8/256/0/180] [15646.047946] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15646.067129] LNet: Added LNI 192.168.204.126@tcp31 [8/256/0/180] [15652.869934] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15652.907844] LNet: Added LNI 192.168.204.126@tcp32 [8/256/0/180] [15660.177399] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15660.218542] LNet: Added LNI 192.168.204.126@tcp33 [8/256/0/180] [15667.871991] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15667.896974] LNet: Added LNI 192.168.204.126@tcp34 [8/256/0/180] [15675.943861] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15675.994388] LNet: Added LNI 192.168.204.126@tcp35 [8/256/0/180] [15683.688185] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15683.713818] LNet: Added LNI 192.168.204.126@tcp36 [8/256/0/180] [15690.461935] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15690.499388] LNet: Added LNI 192.168.204.126@tcp37 [8/256/0/180] [15698.917299] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15698.944601] LNet: Added LNI 192.168.204.126@tcp38 [8/256/0/180] [15708.976111] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15709.040096] LNet: Added LNI 192.168.204.126@tcp39 [8/256/0/180] [15719.503670] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15719.529633] LNet: Added LNI 192.168.204.126@tcp40 [8/256/0/180] [15728.827310] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15728.870035] LNet: Added LNI 192.168.204.126@tcp41 [8/256/0/180] [15739.318506] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15739.355635] LNet: Added LNI 192.168.204.126@tcp42 [8/256/0/180] [15747.894994] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15747.951774] LNet: Added LNI 192.168.204.126@tcp43 [8/256/0/180] [15757.361491] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15757.414659] LNet: Added LNI 192.168.204.126@tcp44 [8/256/0/180] [15766.702249] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15766.728891] LNet: Added LNI 192.168.204.126@tcp45 [8/256/0/180] [15776.120977] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15776.157766] LNet: Added LNI 192.168.204.126@tcp46 [8/256/0/180] [15784.671618] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15784.732163] LNet: Added LNI 192.168.204.126@tcp47 [8/256/0/180] [15794.680512] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15794.718948] LNet: Added LNI 192.168.204.126@tcp48 [8/256/0/180] [15804.808228] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15804.859529] LNet: Added LNI 192.168.204.126@tcp49 [8/256/0/180] [15814.173416] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15814.208963] LNet: Added LNI 192.168.204.126@tcp50 [8/256/0/180] [15833.057328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15833.075436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15833.097463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15835.622846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15835.632247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15835.659493] Lustre: Skipped 1 previous similar message [15838.063107] Lustre: server umount lustre-MDT0000 complete [15842.506592] LustreError: 473954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790726380 with bad export cookie 5007263330127692376 [15842.513082] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15842.517615] LustreError: 473954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15842.957925] Lustre: server umount lustre-MDT0001 complete [15861.216766] Lustre: 473225:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790726383/real 1790726383] req@00000000b662afbd x1877712259181312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790726399 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15861.263416] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15861.289757] Lustre: Skipped 1 previous similar message [15863.266200] Lustre: 473226:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790726385/real 1790726385] req@0000000009086792 x1877712259181440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790726401 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15864.425769] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [15866.337133] Lustre: 473223:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790726388/real 1790726388] req@00000000e71512df x1877712259181568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790726404 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15868.455488] Lustre: 473223:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790726390/real 1790726390] req@0000000011803834 x1877712259181824/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790726406 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15874.528248] Lustre: 473226:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790726396/real 1790726396] req@000000006fe7a027 x1877712259181952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790726412 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15875.072316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15887.904380] LustreError: 473222:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009cec6f4a x1877712259182720/t0(0) o250->MGC192.168.204.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15888.271341] LustreError: 475927:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15888.295166] LustreError: 475927:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [15888.351880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15890.401991] LustreError: 475928:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15890.409108] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15891.426262] LustreError: 475927:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15893.230790] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [15896.546420] LustreError: 475928:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15901.507617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15901.673370] LustreError: 475927:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15901.885243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15906.985455] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15907.352118] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [15916.945360] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15923.075250] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15948.269097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15948.280156] Lustre: Skipped 1 previous similar message [15948.281396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15948.282987] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15948.322441] Lustre: Skipped 1 previous similar message [15952.004461] Lustre: server umount lustre-OST0000 complete [15958.502657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15958.504427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15958.538504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15963.619502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15963.631058] Lustre: Skipped 2 previous similar messages [15972.319481] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15972.686166] Lustre: server umount lustre-MDT0000 complete [15974.882687] LustreError: 500184:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15974.918208] LustreError: 500184:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15977.115326] LustreError: 473953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790726515 with bad export cookie 5007263330127695162 [15977.116409] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15977.129967] LustreError: 473953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15977.559751] Lustre: server umount lustre-MDT0001 complete [15988.692482] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [15991.294229] Key type lgssc unregistered [15991.659417] LNet: 502992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15991.670519] LNetError: 502992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15991.678040] LNet: Removed LNI 192.168.204.126@tcp [15991.680549] LNet: Removed LNI 192.168.204.126@tcp10 [15991.683649] LNet: Removed LNI 192.168.204.126@tcp11 [15991.686865] LNet: Removed LNI 192.168.204.126@tcp12 [15991.689208] LNet: Removed LNI 192.168.204.126@tcp13 [15991.692518] LNet: Removed LNI 192.168.204.126@tcp14 [15991.695199] LNet: Removed LNI 192.168.204.126@tcp15 [15991.698226] LNet: Removed LNI 192.168.204.126@tcp16 [15991.700221] LNet: Removed LNI 192.168.204.126@tcp17 [15991.703528] LNet: Removed LNI 192.168.204.126@tcp18 [15991.706688] LNet: Removed LNI 192.168.204.126@tcp19 [15991.710541] LNet: Removed LNI 192.168.204.126@tcp20 [15991.716127] LNet: Removed LNI 192.168.204.126@tcp21 [15991.719973] LNet: Removed LNI 192.168.204.126@tcp22 [15991.726206] LNet: Removed LNI 192.168.204.126@tcp23 [15991.729477] LNet: Removed LNI 192.168.204.126@tcp24 [15991.731954] LNet: Removed LNI 192.168.204.126@tcp25 [15991.734405] LNet: Removed LNI 192.168.204.126@tcp26 [15991.736423] LNet: Removed LNI 192.168.204.126@tcp27 [15991.740192] LNet: Removed LNI 192.168.204.126@tcp28 [15991.743387] LNet: Removed LNI 192.168.204.126@tcp29 [15991.746918] LNet: Removed LNI 192.168.204.126@tcp30 [15991.751990] LNet: Removed LNI 192.168.204.126@tcp31 [15991.758846] LNet: Removed LNI 192.168.204.126@tcp32 [15991.762651] LNet: Removed LNI 192.168.204.126@tcp33 [15991.765062] LNet: Removed LNI 192.168.204.126@tcp34 [15991.768816] LNet: Removed LNI 192.168.204.126@tcp35 [15991.774177] LNet: Removed LNI 192.168.204.126@tcp36 [15991.777841] LNet: Removed LNI 192.168.204.126@tcp37 [15991.780716] LNet: Removed LNI 192.168.204.126@tcp38 [15991.784334] LNet: Removed LNI 192.168.204.126@tcp39 [15991.787308] LNet: Removed LNI 192.168.204.126@tcp40 [15991.789563] LNet: Removed LNI 192.168.204.126@tcp41 [15991.793790] LNet: Removed LNI 192.168.204.126@tcp42 [15991.797081] LNet: Removed LNI 192.168.204.126@tcp43 [15991.801466] LNet: Removed LNI 192.168.204.126@tcp44 [15991.807871] LNet: Removed LNI 192.168.204.126@tcp45 [15991.812102] LNet: Removed LNI 192.168.204.126@tcp46 [15991.814732] LNet: Removed LNI 192.168.204.126@tcp47 [15991.819641] LNet: Removed LNI 192.168.204.126@tcp48 [15991.823577] LNet: Removed LNI 192.168.204.126@tcp49 [15991.838976] LNet: Removed LNI 192.168.204.126@tcp50 [15992.686302] Key type .llcrypt unregistered [15992.688520] Key type ._llcrypt unregistered [16013.466522] Key type ._llcrypt registered [16013.469178] Key type .llcrypt registered [16013.592092] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [16028.589526] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [16029.867571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16029.905355] alg: No test for adler32 (adler32-zlib) [16031.104686] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [16031.354680] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [16033.071166] Key type lgssc registered [16034.286329] Lustre: Echo OBD driver; http://www.lustre.org/ [16070.176112] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [16070.226588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16071.516898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16071.584072] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16071.652388] Lustre: lustre-MDT0000: new disk, initializing [16071.742844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16071.772587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16076.924857] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16089.174649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16089.299235] Lustre: 506206:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16089.356956] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16089.362207] Lustre: Skipped 1 previous similar message [16089.497396] Lustre: lustre-MDT0001: new disk, initializing [16089.598461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16089.639291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16089.650534] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16095.326445] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16105.990340] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16112.439871] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16122.290028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16122.471549] Lustre: *** cfs_fail_loc=90f, val=0*** [16122.568720] Lustre: lustre-OST0000: new disk, initializing [16122.574981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16122.589109] Lustre: 508224:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16122.713649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16128.122335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16128.136815] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16128.287196] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16130.292843] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16140.927901] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16148.964991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16148.969982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16148.978314] Lustre: Skipped 1 previous similar message [16148.978441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16148.996446] LustreError: Skipped 1 previous similar message [16154.090195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16154.094508] Lustre: Skipped 2 previous similar messages [16154.878144] Lustre: server umount lustre-OST0000 complete [16159.204597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16159.207725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16159.233040] Lustre: Skipped 1 previous similar message [16159.243703] Lustre: Skipped 1 previous similar message [16164.321251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16169.443740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16169.456975] Lustre: Skipped 2 previous similar messages [16173.023253] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16173.267990] Lustre: server umount lustre-MDT0000 complete [16177.349565] LustreError: 507010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790726715 with bad export cookie 1226692592084006006 [16177.361768] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16177.811438] Lustre: server umount lustre-MDT0001 complete [16190.642439] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [16192.817957] Key type lgssc unregistered [16193.017699] LNet: 510132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16193.030846] LNetError: 510132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16193.052971] LNet: Removed LNI 192.168.204.126@tcp [16193.834321] Key type .llcrypt unregistered [16193.839926] Key type ._llcrypt unregistered [16216.787279] Key type ._llcrypt registered [16216.789708] Key type .llcrypt registered [16216.908489] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [16218.139919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16218.247868] alg: No test for adler32 (adler32-zlib) [16219.327508] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [16219.587478] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [16221.295354] Key type lgssc registered [16222.747059] Lustre: Echo OBD driver; http://www.lustre.org/ [16231.296157] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [16231.334188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16232.835797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16237.433326] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16242.174330] LustreError: 511417:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16242.211739] LustreError: 511417:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [16247.188433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16247.265516] LustreError: 511418:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16247.625946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16252.436847] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16259.811283] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16264.816695] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16271.229661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16271.590110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16278.085047] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16285.996920] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16297.096351] loop0: detected capacity change from 0 to 204800000 [16297.353814] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16297.364619] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16297.377278] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16304.112271] loop0: detected capacity change from 0 to 204800000 [16304.302100] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16304.326670] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16304.345877] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16310.057405] loop0: detected capacity change from 0 to 204800000 [16315.976746] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16316.132460] Lustre: fs15246-OST0000: new disk, initializing [16316.136825] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16316.142193] Lustre: 515010:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16316.229914] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16324.024274] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16334.702183] loop1: detected capacity change from 0 to 204800000 [16340.492187] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16340.624742] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16340.683409] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16340.771881] Lustre: fs15246-MDT0000: new disk, initializing [16341.031186] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16341.063229] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16341.203582] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16342.241279] Lustre: 516135:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000291d32fc x1877713099996032/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790726896 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16345.894845] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16351.336298] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16351.349394] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16351.393001] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16370.664205] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16370.703802] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16375.780513] LustreError: 511413:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16375.969851] Lustre: server umount fs15246-MDT0000 complete [16380.915515] LustreError: 511414:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16386.025306] LustreError: 513066:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16388.369450] Lustre: server umount fs15246-OST0000 complete [16400.868903] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16400.869823] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16400.875888] LustreError: Skipped 1 previous similar message [16400.887624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16400.903405] Lustre: Skipped 1 previous similar message [16406.514790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16406.525299] Lustre: Skipped 3 previous similar messages [16411.618023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16411.627203] Lustre: Skipped 1 previous similar message [16415.199928] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16415.500758] Lustre: server umount lustre-OST0000 complete [16421.872451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16421.874417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16421.876295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16421.922452] Lustre: Skipped 2 previous similar messages [16426.373081] Lustre: server umount lustre-MDT0000 complete [16426.981453] LustreError: 511413:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16427.003100] LustreError: 511413:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [16430.804890] LustreError: 511400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790726968 with bad export cookie 3921641540051909541 [16430.816215] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16430.817366] LustreError: 511400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16431.254394] Lustre: server umount lustre-MDT0001 complete [16445.812373] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [16449.073265] Key type lgssc unregistered [16449.444790] LNet: 519236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16449.451967] LNetError: 519236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16449.481518] LNet: Removed LNI 192.168.204.126@tcp [16450.488422] Key type .llcrypt unregistered [16450.491266] Key type ._llcrypt unregistered [16473.689779] Key type ._llcrypt registered [16473.692450] Key type .llcrypt registered [16473.844590] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [16475.340506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16475.488358] alg: No test for adler32 (adler32-zlib) [16476.706193] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [16477.114682] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [16478.831304] Key type lgssc registered [16480.125505] Lustre: Echo OBD driver; http://www.lustre.org/ [16527.853246] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [16539.589482] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [16586.266239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [16586.323804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16587.876871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16587.968660] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16588.127352] Lustre: lustre-MDT0000: new disk, initializing [16588.265815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16588.294267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16593.676209] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16605.625820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16605.830285] Lustre: 523724:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16605.946359] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16605.952285] Lustre: Skipped 1 previous similar message [16606.127331] Lustre: lustre-MDT0001: new disk, initializing [16606.300666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16606.385294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16606.395263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16612.339180] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16623.927723] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16630.165477] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16637.874459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16638.216842] Lustre: lustre-OST0000: new disk, initializing [16638.222593] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16638.234583] Lustre: 525694:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16638.353733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16645.219617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16645.232803] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16645.310795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16645.456268] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16656.435062] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16661.130943] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16661.453701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16666.787964] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16667.106875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16671.201504] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16671.213602] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16671.245917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16671.255573] Lustre: Skipped 1 previous similar message [16675.518365] Lustre: server umount lustre-OST0000 complete [16681.444477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16681.464971] Lustre: Skipped 1 previous similar message [16681.472111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16681.483113] Lustre: Skipped 1 previous similar message [16686.479729] Lustre: server umount lustre-MDT0000 complete [16691.203248] LustreError: 524530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790727229 with bad export cookie 18128840219910518483 [16691.206888] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16691.214690] LustreError: 524530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16691.747869] Lustre: server umount lustre-MDT0001 complete [16721.562369] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [16733.101765] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [16770.818432] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16460 sec ========= 20:15:08 (1790727308) [16773.164637] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:15:10 (1790727310) === [16777.229541] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:15:14 (1790727314) === [16823.077311] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [16825.848461] Key type lgssc unregistered [16826.118846] LNet: 532340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16826.128476] LNetError: 532340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16826.141726] LNet: Removed LNI 192.168.204.126@tcp [16827.099769] Key type .llcrypt unregistered [16827.105893] Key type ._llcrypt unregistered